Electronic – Why would an AC motor heavily shake when driven with certain frequencies

All digital ICs should have decoupling caps between their power and ground pins. These should be ceramic and physically as close as possible to the IC. You want to minimize the loop length from power pin to cap to ground pin thru the IC and back to the power pin.

Decoupling caps deal with short term current spikes the IC draws. They must therefore be high frequency. Large capacitance is not necessary, and since large capacitors usually have poor high frequency response, they are worse. A 100µF electrolytic cap is pretty useless for decoupling. 1µF to 100nF ceramic is good.

As for the cap on the motor, the idea is good but I think 100nF is too large. That could cause excessive or unnecessary current to flow in the H bridge every time it switches. If you're only reversing motor direction occasionally, then this isn't a big deal. If you're using the H bridge to modulate the apparent motor drive with PWM, then you should lower the cap. Something like 1nF should still cut down the noise the motor is making while not getting in the way of switching.

Andy beat me to it. But anyway. I am a power electronics specialist - that sound you hear is the harmonic noise being caused by the square waves you are passing into the motor. It is physically damaging the motor and reducing efficiency by creating heating losses in the coils that don't create useful work, because they occur at other frequencies than where the work can happen.

This is a basic two level sawtooth modulated PWM signal, just for reference.

See how the PWM signal is high much longer at the peak of the sinusoidal wave? That's where the magic is. If we divide it up into enough pieces and put well selected low-pass filter on it, it will practically turn into a pure sine wave.

Well, that's not really true. The voltage will still be blocky but the current waveform will have so much less distortion that you no longer need to worry about life-shortening harmonics and ear-grating noise.

You're already halfway there in many ways. If you're already generating a duty cycle now, with very little effort you will be able to find a library to bit-bang this kind of PWM signal output. Depending on the exact controller you are using, it may actually have an onboard function for this very thing.