I think if you haven't got bypass caps present, then this is very likely to be the problem (especially given the RF isolation)
The datasheet recommends 1uF ceramic caps be placed on both sets of supply pins. Put them as close as possible to the pins.
Also, make sure you have the pullup resistors present on both sides.
If you have an oscilloscope, you should be able to test the lines easily (e.g. send repeated signal and see if what's going in one side is coming out okay the other)
Well, the quick answer is no. That won't work.
1) Your flyback diodes are pointing the wrong way, and will always be ON, shorting out the power supply. They are also not rated for 1.2 amps.
2) Your optocouplers, while not a bad idea, cannot possibly handle the current required.
3) It's not at all clear what your logic function is, and I suspect you intended U1,U3,U14,U18 and U22 to be PNPs.
Since you're willing to use external logic, and apparently don't entirely trust your Pi to provide only one output active at a time, I would suggest you go whole hog on logic (it will only take one IC) and do something like
simulate this circuit – Schematic created using CircuitLab
D0 to D2 are 3 bits from the Pi, and they form a binary address of the motor you want on. A code of 1 to 5 will select the motor you want, and 0,6,or 7 will result in no motors on.
I've shown the 5 optocoupler inputs as LEDs, and I've only shown one output. The flyback diode should be rated for 2 amps or more. A 1-amp diode such as a 1N5817 will actually work, but whenever a motor is turned off the current through the diode will briefly exceed 1 amp. This is probably OK, but bad practice. MOSFETs are n-type. Almost anything will do, since voltage rating is less than 20 volts, and current is 1.2 amps.
Best Answer
I was thinking about this same thing yesterday and came up with this thread when I searched. I know there isn't much need to re-invent wheels, but it is a fun design problem, and who knows, one day we might need something in a hurry and have to make something from the spare parts drawer. So - what about this? I think it does the job. Only one open-collector line show, the two are the same. One dual-opto per line.