How's this for a theory? Shoot it down in flames if you like...
The spinning magnetic field inside the motor (be it a spinning electromagnet in a brushed motor or a spinning solid-state magnet in a BLDCM) induces an alternating electric field into the metal body of the motor. The average potential of this field is zero, but the instantaneous potential could be quite high. The capacitors are there to leech this electric field away from the body before it can radiate out from the body, thus reducing the possibility of it interfering electromagnetically with the surrounding delicate electronics.
The ground clip of your scope is connected to Earth. If anything else in your motor drive circuit is connected to Earth, this can lead to some large currents through the scope, which is noise at best, burnt fuses, traces, smoke, fire at worst.
Even if your motor drive circuit is floating, by clamping one half of the motor to Earth, you make the circuit unsymmetrical. Everything in your circuit has some capacitance to everything else in the universe, Earth being a pretty significant part of that. To change the voltage across a capacitance (such as that between any part of your circuit and Earth) some current must flow, according to:
$$ I = C \frac{dv}{dt} $$
Normally this isn't a big deal, but I'm guessing you have some relatively high voltages involved, since you have a node labeled "HV_BUS" and that's a 500V MOSFET. Normally, it's just the MTR- node that changes voltage relative to Earth significantly, and these capacitive currents are really insignificant compared to the motor currents. But when you force MTR- to be at Earth potential, now it's everything else in your circuit that's switching \$\pm500V\$ relative to Earth, and these capacitive currents are in your ground, in your control circuitry, etc.
For a simpler demonstration of problems introduced by capacitive coupling to Earth, see Why one of my multimeter shows a small voltage when only one probe is connected to AC and the other dont?
The way to work around this is to clip your scope probe to the circuit ground. If you need to measure the difference between the motor terminals, use both channels on your scope, each clipped to ground and one of the motor terminals, and use the X-Y difference function of your scope.
Or, since HV_BUS is a constant voltage from ground, you can clip your scope ground lead here only if the motor driver is floating and you are confident this won't introduce a safety hazard and just consider the voltages you measure to be negative.
Best Answer
100nF was and still is a very common value at 50VDC . The precise value is not important so ceremic capacitors are used . They are cheaper and have lower ESR so they actually work better than the physicaly larger more expensive metal film caps .These caps were decided on before larger ceremic caps in the microfarad range became cheap.Larger caps will have a lower self resonant frequency especially when considering the practical lead lengths in a motor.This could make big caps useless at suppressing radiated EMC.So using huge caps on a huge motor is not the answer . Feedthrus are often used for high currents.Most orthodox PWM controllers wont like a large cap across the motor terminals because the mosfet would be turning on into it .