PIC resets when motors work

The usual suspects:

1. Make sure MCLR is held solidly high, or that the MCLR function is internally disabled. I don't remember if MCLR can be internally disabled on the 18F4520.

2. Make sure the PGM pin is not floating, or that low voltage programming is disabled.

3. Make sure there are bypass caps on every power pin, and that they connect back to a near ground pin with as short a loop area as you can manage. All the ground pins must be tied together, prefereably right under the PIC.

4. Make sure all power and ground pins are connected. This includes the analog power and ground, even if you're not using any of the analog peripherals.

5. Either make sure your power voltage is really clean or disable the brownout reset feature. The brown out detect circuitry is less than ideal on some PICs, so the first thing to do when you suspect something like this is to disable brownout reset in the code and see how that changes the symptoms. I have seen fast positive glitches on the power trigger the brownout detect, for example.

6. Decouple the digital and high power supplies. At the very least, put a Shottky diode in series with the digital supply so that short dropouts due to sudden power load changes don't drop the digital supply too. Of course the voltage will have to be adjusted to compensate for the diode drop. Depending on the supply, you may be able to put the diode within the voltage control feedback loop.

7. Don't forget the reverse diode accross any coil, which includes any motor winding. If the coil is driven bi-directionally, then make sure there is a fast diode in reverse to at least the power and ground on each coil lead.

8. Pay attention to the ground layout. Preferably this is a plane or mostly a plane with occasional "jumpers" that connect points on the other layer. Keep high frequency currents, like those between the PIC and its crystal, off the common ground plane. Use a local ground for the PIC and it's immediate surrounds, then tie that to the main ground in one location. Remember that everything you do to minimize radiation works the other way too and helps with susceptibility.

While circuit topology, like forgeting inductor catch diodes, can cause problems like this, after all the really stupid stuff the most common cause is bad layout.

My guess (there's not really enough information for more than a guess) is that the 5V supply rail is collapsing when the motor turns on (motor may be stalled, too powerful for your supply, etc.). To troubleshoot this further, break the problem down:

First, disconnect the PIC and the motor and just focus on the driver chip. Make sure ALL the inputs are defined (pulled high or low) - even the inputs you aren't using. Make sure any outputs you aren't using are floating. Now, with 5V on the Vcc of the driver, take the EN pin high, toggle an A input, and verify that the appropriate Y output goes high and low following the A input. Do that for both A inputs. If it doesn't work, record the voltages on ALL the pins (supply, input, output, even GND in case you miswired something) of the driver and report them here.

After that's working, connect the motor, toggle the A inputs again and verify that the Y outputs change as before and that the motor turns as expected. If it doesn't, record the voltages on ALL the pins of the driver (particularly the Vcc pins) and report them here.

Next, connect the PIC, and repeat the above procedure with the PIC driving the pins.

If you go through it methodically, you will find the problem.