When using high-side N-channel MOSFETs, a MOSFET driver chip is generally used (e.g., an IR2130). These chips require a boost capacitor which allows the gates of the high-side MOSFETs to be driven with the required voltage. The downside of this is that the MOSFET driver has a restricted operating range (typically 10V-20V). The second down-side is that you can't drive the MOSFETS to 100% pulse-width; this is usually not a problem. Alternatively you could use P-channel high-side MOSFETs and N-channel for the low-side. With this configuration, you can PWM the low-side switches only and save some power. (You can do this with all N-channels as well, it's just trickier since you have to keep the boost capacitors charged). Go to Digikey and search for "mosfet driver", you'll want the 3-phase bridge drivers specified for external switches.
As for the MOSFETs themselves, I don't recommend you use them in parallel. This would increase your footprint as well as your gate charge. Instead, look for high-current mosfets on Digikey. My favorite packages for these are D-Pak, 8-PowerVDFN, and PowerPAK 1212-8.
Atmel has an app note for sensor-based and sensor-less motor control. Unfortunately AVRs don't have much processing power and your control algorithm will likely be restricted to fixed-point math because of this. Because of the processing power limitations, you may want to consider not using the Arduino environment and instead use straight C or assembly.
You may have some trouble isolating the I2C bus optically since it's a bi-directional bus. If you were to use a communication system with single direction lines, you may want to look at TI's line of isolator chips (e.g., the ISO7221).
I'd like to place these additional CPLDs on a different PCB. This has the advantage that I can simply extend the device when I need. On smaller harnesses, a single 114 test-point PCB will suffice. On larger ones, I can cascade.
There are multiple levels of modularization which you can aim for. Where you want to stop depends on your specific use case. At the most basic level, the hardware must be designed such that you can select the number of modules in use after the design stage. The difficulty of changing the number of modules, space available, desired software complexity (and available space for software, especially on a CPLD) and the system cost will be key factors in your decision.
Hardware
The simplest and cheapest way to do this is to build one PCB, (You don't need multiple PCBs for modular design!) and put footprints for your desired maximum number of CPLDs on the PCB. If you need more IO, you can then solder down another CPLD. Obviously, this isn't something you'd want to do very often.
At the next stage, you'd want to build daughtercards so that you can more easily add and remove modules. You asked:
But what would be the best way to actually connect the PCBs together?
This depends on your system architecture and number of modules. If you know you'll never want more than, say, 3 modules at any one time, just put three connectors on the main board. These can be edge connectors, or stackable connectors, or whatever you like that doesn't require wires. If the number of sub-modules is too large to fit connectors for each on one PCB, then you should consider stacking (if your bus can handle the fanout of your maximum number of modules) or daisy-chaining (if you need to buffer the signal or vertical space is limited) the modules.
Plenty of connectors are designed for this purpose; check the "Board-to-Board" section of your favorite distributor or manufacturer, and many are designed for extremely low crosstalk and high frequency - 500kHz is nothing, unless you're using PTH 0.1" breakaway headers and have fast-changing signals (even then, you're probably OK). Check the mating strength of your connectors just to be sure, but if you only have a few pins, the footprint of your interconnection doesn't carry the stresses well, or the system will be subject to vibration, you'll need standoffs. It's often wise to design the interface in such a way that different modules can be designed to interface with the motherboard in the future. Pins are cheap, give yourself a couple spares just in case!
Software
If your number of modules supports it, you can simply add a slave select line for each module. This isn't really a software solution, but I wanted to mention it.
If you don't mind programming every CPLD differently, you could build the system such that the microcontroller sees it as one giant shift register (which you've suggested). If you added or removed a module, that module's address space would simply be wasted, and extra time would be used transmitting to addresses which don't exist. Each module would need to 'know' its address space, though, which would make programming the complete system a struggle.
A more versatile solution is to use software addressing to access each sub-module. In a 'programming mode' (perhaps a pushbutton on the module, or simply only connecting one at a time), you could assign the CPLD an address. By assigning each CPLD a different address, you could add or remove modules at will, and only have to adjust the activity of the microcontroller (which I presume to be easier to adjust than the CPLD).
My suggestion for this project
If a 324-pin device will solve all your foreseeable use cases, then the single-PCB method should work fine. The multiple-slave-select method would allow you to program all the CPLDs simultaneously with a single programmer. Sorry, but this project as described doesn't seem like a candidate for daughtercards.
Best Answer
Consider using a flex circuit
FH12A-12S-0.5SH(55) with 12 pins is one I'm using now. This series has 0.5mm pitch.. 10 pins is less than 10mm long even with the end hinges.
You need to specify the FPC with a stiffener that meets the connector thickness requirement and to make the shape tolerances compatible, other than that it's pretty painless.
The lever cover makes connecting a snap.
There are vertical types too.