Servos usually use a simple protocol based on pulses. Servo boards are designed to generate these pattern(s). DC-Motors are usually driven using a H-bridge (wikipedia), which uses a (constant) controlled current flow. So technically it is possible, but not really practical to use a servo board to drive a DC motor.
I think you will find that the "0V" portion is not really 0V but actually -0.7V (you can see on the image it is somewhere below 0V, check the scope setup to be sure) and that is the back-EMF spike from the motor's inductance turning on the MOSFET's body diode.
V = -L * dI / dT
As the body diode limits the voltage (a good thing, it could be a few hundred volts otherwise, and destroy the transistors!) the pulse width lasts until the current has fallen to 0. This pulse width will depend on the motor inductance and the current; it probably increases as you load the motor.
Then, with both transistors off, you see the voltage generated by the motor. The only way to eliminate this is to stop the motor - it is proportional to speed. So again, as you load the motor you will slow it, and see the reduction in generated voltage.
You could even sample this voltage during the PWM "off" period to measure the actual motor speed.
The initial peak and decay on the "high" portion of the waveform is again related to the motor's inductance - when the MOSFETs first turn on, the inductance presents a high impedance, so the supply is lightly loaded and the voltage across the motor is high. As the motor current increases, the voltage drops.
If you can see this peak and decay on the incoming power supply, you can improve performance by adding decoupling or using a lower impedance PSU. If you can't, that suggests some voltage is being dropped across the MOSFETs, and you can improve performance using FETs with lower ON resistance.
Best Answer
Your circuit is on the right track. Applying a resistor in parallel with the motor will work. The voltage produced by the motor depends on the direction it spins. It won't reverse just because you stop applying power.
The motor will keep the switch node at a voltage between the supply voltage and ground. At this point, shorting the motor or applying a resistor across its terminals will allow it to drive current opposite the direction it normally flows dissipating energy into the resistor and/or its own internal resistance.
Using an N-channel MOSFET is problematic. It should be either on (Vgate=10V) or off (Vgate=0 volts) as you have it now, the circuit applies whatever voltage is across the motor to the gate. As the motor slows down, this voltage will decrease and the MOSFET will eventually enter it's linear region. At that point, it will dissipate more and more of the braking power and will POP. You need some way of supplying a voltage higher than the supply voltage to keep the MOSFET fully turned on. This can be done with a bootstrap circuit possibly with a charge pump depending on how exactly the brake will be used. The alternative would be to use a P-channel MOSFET on the high side to switch the braking resistor into the circuit. This simplifies gate drive.
Of note, the triangle wave generator in the controller generates a square wave internally that could be used to drive a charge pump.
Here's what a charge pump based solution would look like:
Note: the Zener diode should be 10-12 volts. It limits the voltage produced by the charge pump. The capacitor should be large enough to supply the current required by the turn off resistor while keeping the gate drive voltage at a reasonable level.
Here's what doing the same thing with an P-channel Mosfet would look like. Notice how much simpler the gate drive is. The resistor ratio determines the voltage ratio between the signal and the gate drive voltage.:
P-Channel MOSFETs may be a little more expensive than equivalent N-channel MOSFETs but not that much more. It's worth considering them for this application.
Here's how to do it with a relay: