I don't know how you came up with needing 5 V, but it sounds like a bad idea. You have a big efficiency problem, so spending a bit more on power electronics will make things easier and cheaper in the long run.
First, I would not bus around power as low as 5 V because that will require too much current. Having a roughly 48 V bus sounds like a much better idea. Each panel can then locally convert that to the specific voltages needed to run the LEDs and the electronics. That also gets around the problem of the bus voltage drooping from the power supply to the panels and between panels. The buck supplies on each panel can tolerate significant variations in the "48 V" power bus. And, because of the lower current there will be less variation in the first place.
Look at what voltages the LEDs need. Red and green will be near 2 V, but blue over 3. Red and green are probably close enough so that you can use one power voltage for both of them. Green has the higher voltage, usually about 2.1 V, so make a little more than that. You want it only high enough so that you can put enough of a resistor there to have the current be reasonably predictable despite variation in the LEDs. Maybe 2.5 V is a reasonable tradeoff. Red LEDs usually drop a bit under 2 V, so the regulation for red will be slightly better. Either way, this is still way better than 5 V. For the same LED brightness, just switching to 2.5 V instead of 5 V will save half the power.
Blue usually requires significantly more voltage, like over 3 V. Make a separate supply for blue. It should be a few 100 mV above the LED voltage, just like for the red and green LEDs.
48 V is a common voltage for off the shelf power supplies, and is the limit you are usually allowed before you get into legal regulations. There are various buck converter chips out there, or if you're clever you can maybe have a existing micro handle the buck conversions. Either way, these are readily available blocks you can use in your circuit.
I would suggest doing this in a small FPGA. What you want is clockwork it is more suited to an FPGA than a CPU. The VHDL for it is trivial.
You can get cheap FPGA dev boards, e.g. From Actel (Igloo Nano) that have all the hardware on board that you need. You just need to connect a level translator for the MIDI and dot matrix driver, e.g. those given in other answers. The dev board has pins to do that. Instead of a PCB you can even use stripboard. I've connected an LCD matrix display in this way.
Igloo Dev Boar Manual
Best Answer
1) LED lifetime is related to thermal, the higher the temperature the shorter the life. Make your LED's happy by giving them less current and/or cool them off.
2) If your worried about voltage spikes, then find a zener diode and clamp the voltage across C4. Make sure the zener is going to be able to handle the spike and not blow up.
3) If your LED's have significant voltage drop differences (some have a 3.3V drop, and some have a 3.6V drop) some of them could be dissipating much more power than others. If this is happening, you will have to match them.