I don't know what you mean by "connected" your headphone amplifier to USB. I'll assume you mean that you are powering it from the USB. In that case, the noise on the USB power is getting into the headphone signal.
Since the amp apparently isn't designed to filter the power input, you'll have to add that. Put a small inductor in series with the USB power line, then add a capacitor to ground on the amp side of the inductor. You probably don't need much of a inductor. Even a ferrite "chip inductor" would probably help. The most important point is to put a few 10s of µF of ceramic capacitance immediately accross the power input to the amp. A few 100 µF electrolytic in addition couldn't hurt.
Grounding can be tricky for noisy circuits. Switching power supplies and digital stuff both create noise. Put those two concepts together, and you have the problem you described.
Basically, you're seeing Ohm's law on the ground traces/wires making it not ground anymore at some distance away. Because that trace/wire is being used as a reference for your signal, any noise on the reference is interpreted as signal also. If you had a dedicated reference instead of sharing the power return for that purpose, then you could subtract out that noise using a balanced receiver. This is how balanced cables work: they're all about equal pickup of noise and nothing about opposite signals. (https://sound-au.com/articles/balanced-2.htm)
Ground loop isolators typically use transformers to fix the problem a bit differently, by converting the signal from electric to magnetic and back again, which allows the signal to pass with no electrical connection. The cheap ones, however, may not be able to handle low frequencies at high volumes without distorting because the iron core that makes them more efficient can also saturate. This is the magnetic equivalent of clipping. (The really cheap ones may actually be power transformers in a different package. Same concept, and sorta works, but different design criteria.)
Or, as your experiment showed, you could use two isolated supplies for the parts of your circuit that don't like each other, and connect their grounds at exactly one point, that point being the reference for your signal. This prevents any current from flowing through that reference so that it's only a reference and not part of a power supply.
So, I see three ways to fix your problem, which may be combined if necessary:
- Use a balanced connection from the receiver to the amp.
- Use a (good quality) ground loop isolator or audio transformer.
- Isolate the power supplies from each other and reconnect them only through the audio cable. If you don't want separate 110v adapters, you can use an isolated DC-DC converter instead.
Best Answer
After I provided a separate power supply through a separate onboard DC DC isolator the noise went away. Earlier I was trying to power the bluetooth module from the 3.3v voltage regulator of Teensy which was creating the noise problem. So bluetooth modules need a ground isolated power supply independently. Even if one takes a isolated power and send it through a voltage regulator of an arduino it will create problems. So a separate DC DC isolator is needed.