According to the data sheet, the 3 RGB LEDs are electrically separate. This means that you can connect them in series, using a higher voltage with fewer resistors and transistors. Nick Alexeyev's answer then applies. Assuming a 36 volt power supply, and strings of 8 for green and blue, 16 for red, and 24 for IR, total is 18 channels. I would not go with Nick's suggestion of 48v/12x strings for green and blue, since there isn't enough excess voltage for the limiting resistors to operate reliably, particularly with the Vf variations given in the data sheet. I'd expect that you'd need to measure the voltage drop of each string and tailor the limit resistor values accordingly.
What I think you've missed is power. Assuming 20mA for each LED, total power is 3.6 watts each for green and blue, 2.3 watts for red, and 1.5 watts for IR. Total power is 11 watts in the LEDs. I have no idea how you're going to heatsink this. Well, I do, but it involves using a beryllium oxide substrate for your LED PC board, bonded either to a pretty hefty heat sink, or maybe a TEC cooler. You want the LEDs to run as cool as possible for better lifetime. But trying to do it with FR4 is asking for early death of your LEDs. Similarly, you would also need to calculate the dissipation in your limiting resistors, although for the values I've given I'd expect total dissipation in the 4-5 watt range, and this can be handled with forced air cooling. And with the cooling requirements indicating a certain amount of increased size, I don't think you really need to worry about minimizing the driver board size, although at 18 channels you shouldn't have much difficulty.
I am not aware of any driver ic with that number of outputs. But there are solutions. If you don't need fast refresh rates or high pwm resolution, you could use a microcontroller as a driver. you can get a beefy one and code a pwm algorithm that will use one timer and as many gpios as possible to drive the mosfets that will switch your leds on and off.
I work with PICs and the 122pin pic32mz comes to my mind. Attach many of them as slaves with different adresses in an i2c bus and you are in business.
The adresses can be set with dip switches so you don't have to write different code for each of the uC.
Whatever you chose won't come cheap though at this number of leds.
If you provide us with the displayed material and the refresh rates I can give you a detailed answer with a more detailed analysis as far as timing and cost is concerned.
By Using Transistors. Or Mosfets. The led Driving pins are basically Open Drain, so a pull up resistor and a transistor, with a low base voltage as to meet the WS2821's regulating Voltage Limit (20~40mA at ~1V, more than that can cause heat issues), would work. I've seen the same thing done on TI LED drivers. Inverted logic of course.