Electronic – Using MOSFET to control positive side of motor

Your circuit was connected incorrectly:

Pin 1 is the one nearest the large circular mark, at the left lower side when you hold the part with the printed text correctly oriented and facing you. Pins 2, 3 and 4 go anticlockwise from there.

The Omron MOSFET relay datasheet indicates that your control current needs to be flowing between pins 1 and 2, with positive going to pin 1. Pin 2 will go to ground. Reversing these two pins will not work.

The load is switched by wiring it via pins 3 and 4: Thus 3 and 4 are wired in series with your load, like a push-button would be. The current flow on the switched side (pins 3 and 4) can be in either direction. For convenience, connect pin 3 to Ground, it being closer to pin 2.

Ensure that the motor does not draw more than 1.2 amperes even at startup or stall conditions: The "Recommended Operating Conditions" section states 2 amperes, which realistically works out to 1.2 Amperes at maximum recommended operating temperature (65 degrees C). This rating reduction is as per the specification for derating at 20 mA per degree Centigrade over 25 degrees.

It is worth noting that when a motor or other inductive load is switched off, a fairly high reverse voltage spike can develop across the switch contacts: Typically clamping or bypass diodes are connected to safely drain this spike so it does not damage the MOSFET switches. Many MOSFET packages integrate these diodes internally, but it is still worth adding external diodes in case that's what is "smoking" your relays.

Voltage is a potential relative to a point. In electronics, most of the time that point is circuit common, or ground. When the grounds aren't shared, there is no reference, so the 5V coming from the Arduino is meaningless.

Another thing, put the common lead of the power supply as close as possible to MOSFET return path, ideally in the pin next to it. That will keep the return current off your ground bus, and that is known to make microcontrollers behave in undesirable ways. It may not be a problem for this circuit since its only 120mA, but it's a good habit.