RGB LED matrix power too low

According to the data sheet, the 3 RGB LEDs are electrically separate. This means that you can connect them in series, using a higher voltage with fewer resistors and transistors. Nick Alexeyev's answer then applies. Assuming a 36 volt power supply, and strings of 8 for green and blue, 16 for red, and 24 for IR, total is 18 channels. I would not go with Nick's suggestion of 48v/12x strings for green and blue, since there isn't enough excess voltage for the limiting resistors to operate reliably, particularly with the Vf variations given in the data sheet. I'd expect that you'd need to measure the voltage drop of each string and tailor the limit resistor values accordingly.

What I think you've missed is power. Assuming 20mA for each LED, total power is 3.6 watts each for green and blue, 2.3 watts for red, and 1.5 watts for IR. Total power is 11 watts in the LEDs. I have no idea how you're going to heatsink this. Well, I do, but it involves using a beryllium oxide substrate for your LED PC board, bonded either to a pretty hefty heat sink, or maybe a TEC cooler. You want the LEDs to run as cool as possible for better lifetime. But trying to do it with FR4 is asking for early death of your LEDs. Similarly, you would also need to calculate the dissipation in your limiting resistors, although for the values I've given I'd expect total dissipation in the 4-5 watt range, and this can be handled with forced air cooling. And with the cooling requirements indicating a certain amount of increased size, I don't think you really need to worry about minimizing the driver board size, although at 18 channels you shouldn't have much difficulty.

I believe your issues are not hardware but in the software you use. The reason you get periodic strobing or inconsistent brightness between rows while multiplexing a LED array is because the on time for each row is not consistent. Even though your eye cannot see the row turning on and off because it is happening faster than your persistence of vision allows, your eye can pick up differences in total on time. This is the basic principle behind enabling PWM-based gray scale on LEDs. In order to fix things, you need to ensure that the on-time for each row is consistent.

The core issue is that delaying 100 microseconds between rows is not the same thing as updating each row every 100 microseconds. The amount of time to update a row with your code can vary. For example, if you update row 1, then delay 100 microseconds, then it takes 200 microseconds to update row 2, row 1 has been on for a total of 300 microseconds. If it takes 50 microseconds to update row 3, row 2 would have been on for 150 microseconds. You can then see how using a delay between each row update can lead to inconsistent on-times if each row calculations are themselves inconsistent.

In your code, there are several things that would lead to inconsistent on-time between rows. First, you use a 10-level if-then block to determine the row control bits. That is going to take longer to evaluate on row 10 than on row 1 and is likely why your bottom rows are brighter than your top rows. Second, a similar effect will issue in your setBitMaps() function when evaluating behavior differences between cycles. I suspect this amplifies the last-row brightness. Finally, you are effectively creating your own loop within the overall AVR loop. When you exit your own loop and then do the main AVR loop again, you completely recalculate the bitmap for your LEDs. I suspect this is causing the occasional stabbing as it will take more take more time before you re-enter your internal loops that does the cycle-scans, meaning that while you are recalculating the bit map, the LEDs are left on in the last configuration, hence a momentary flash of brightness (strobe).

I made a 8x8 LED matrix of similar design and wrote a driver which is available on GitHub here. My driver is able to create a stable (non-flashing) image. While my driver is a bit more robust than yours, the key difference is that I am using a timer interrupt on the AVR micro controller. Outside of the interrupt I do all the calculations need to determine what the bits should be for each row, and when the interrupt fires, all I do is blast the next rows bits out to the shift registers. I use SPI rather than bit-banging, but the general concept is still the same. What the timer interrupt provides is a consistent start time for each row update, and only doing the bit transmission during the interrupt makes the work done very consistent, thus the row updates (when the latch pin is fired) are very consistently timed, resulting in a smooth image.

Another key element of my design is that I have two bit buffers that contain the row-bits for a given image frame. The row-by-row bit dumping occurs from one buffer, while the calculation for the next image frame is done on the other buffer. When the next image frame is ready and the row scan is restarting, I simply swap the buffers. This helps focus the work needed during the timer interrupts to only being the row bits transmission to the shift registers.