I'm interested in wiring up some Raspberry Pi Compute Modules together on a custom PCB board with DDR2 SODIMM connector interfaces.
http://www.raspberrypi.org/raspberry-pi-compute-module-new-product/
I want to use the compute modules together in a N-body-type simulation where each module acts as a planet or something and interacts with all other modules via MPI. Ideally each module should be able to asynchronously communicate with any other module in the board without interrupting the processing on the other modules.
What is a good hardware solution to high-bandwidth switching for PCB components connected in this fashion? Normally with standard Raspberry Pis I would use an ethernet switch to route packets between the nodes but I'm not so sure here. Would it be feasible to build switching logic + processors directly into the board (or have several of the compute modules serve as switching logic)?
Some research seems to indicate that Blue Gene node cards (it has a similar design to the tray I am proposing above) are connected via tx+rx 12-channel optical fibers, but I'm not sure if that is what I need, as that seems to be missing some kind of switching capability.
Thanks!
Best Answer
How many modules are we talking about here? 10? 10,000?
There are a couple of possibilities, depending on what sort of I/O capability the boards have and how many boards you want to connect. I'm not aware of any obvious off-the-shelf solutions, however.
One method is to provide each module with one or more Ethernet PHY chips, and then use some commodity ethernet switch chips to connect them together, presuming the compute modules can interface well with a PHY chip. This could work well if a non-mesh network is OK (I don't think el cheapo ethernet switches can handle loops). If you want to do something a bit more complicated, you could make a bunch of small boards with a handful of compute modules, PHY chips, and an integrated switch, then connect the boards together with a larger off-the-shelf Ethernet switch. You would probably be limited to using gigabit ethernet, though.
Another possibility is to use some FPGAs. Inside of an FPGA there is a decent number of block RAMs that can be configured as dual port RAMs or FIFOs. If you have a relatively small number of compute modules, you can just connect them all to a single large FPGA with enough pins, then put togther an FPGA configuration with all of the interface logic as well as a bunch of interconnecting block RAMs and/or FIFOs. If you have a large number of modules, then what you can do is use FPGAs to build a mesh network with a small number of compute modules connected to each FPGA. This way you have complete control over the routing of data between modules. I would recommend using either parallel interconnections between the FPGAs or possibly serialized LVDS for more bandwidth, depending on if the FPGAs have transceivers and how crazy you feel like getting with the board layout.