Electronic – Mains hum without ground loop

I don't know what "earth" you've used for the first experiment, so I'll [mostly] punt on that (for now). I suspect you have a SMPS with an Y-rated capacitor connecting its isolated side with the primary. This is permitted provided that leakage current is kept under control. It's done to reduce EMI from the power supply. You can look at this paper for example. Diagram from a better quality article

The tradeoff is that

In general, the larger the value of the Y-capacitor, the lower the magnitude of the EMI that the supply generates. Conversely, the larger the value of the Y-capacitor, the higher will be the leakage current that flows across the isolation barrier.

Now onto your question:

If I connect an unterminated 2 meter unbalanced ¼” jack cable to one of the inputs, I can hear lots of 50 Hz hum on the output. This is not surprising as the cable is acting as an antenna and the 0404 has high input impedance. However, when I connect the sleeve part of the jack to earth, most of the hum disappears. I can still increase the hum significantly by touching the tip of the jack with my finger.

Yes, the shielding does what it's supposed to on the cable itself. The 50Hz hum signal is low frequency, so it gets coupled into the shield and shorted to signal ground. This is electrostatic shielding i.e. negates capacitively coupled signals by shorting them back their source:

But you are not shielded and you are also capacitively coupled to the mains... as hard as that may seem to believe. (I'll add some youtube videos links about this once I find them. There sure must be some. N.B. Here's one: https://www.youtube.com/watch?v=c3SRuX9lW58) So you negate the effect of the cable shield when you touch the tip of the jack.

If I remove the earth connection and insert the jack into a shielded but unearthed box, I can no longer detect the hum with my ears.

You need to clarify this experiment as well. How can you insert just the jack in a box? Do you mean the whole 0404 USB? Add a photo you can. Shielding without a ground (i.e. sans current return) connection is effective against electromagnetic (but not against electrostatic) interference/noise. I doubt however that simply shielding the jack can have the effect you describe, especially against low frequency noise like the mains frequency. I suggest this app note for more in-depth treatment of shielding (both electrostatic and electromagnetic). See also this answer and yet another app note that says not to bother much with anti-electromagnetic cable shielding as it's hard to achieve (instead simpler things like twisted cable pairs are more effective at negating that).

Ok, a few things.

1. The audio is transmitted digitally, so unless the digital audio you are piping to the RN52 itself has this left channel hum, then then problem is occurring somewhere between the RN52's analog output pins, and the speaker.

2. Wire insulation protects against accidental shorts only. How much of it is removed from where, or if it is even present at all, is completely irrelevant to what is conducting through the wire, audio fidelity, signal integrity, or anything else. If it was shielded wire, that would be a different story. But shields are conductive, and require a second connection. Plain old insulated wire could be replaced with a coat hanger and it would not impact anything (as long as you were careful not to short anything to it, obviously).

3. Poor conductivity causes signal loss (signal is lower amplitude/quieter). It does not create noise or hum out of thin air. As long as the wire is soldered correctly with proper wetting and sufficient solder to be wicked up by the wire and over the entire pad surface, the connection is more than adequate. Make sure you don't have any solder bridges causing shorts, and that you've cleaned off the flux (I like to use isopropanol + a tooth brush) and get a proper continuity checker to check. Otherwise, you don't need to worry about how much insulation you removed from a wire tip, or poor conductivity or anything like that.

4. And now, the actual answer. The RN52's two speaker output channels are differential audio output only, which is referred to in an audio context as balanced audio. This is different from most audio signals, which are typically single-ended, or unbalanced audio, and you cannot simply connect balanced audio signals to anything else unless it is specifically designed to use balanced signals. And unlike unbalanced audio, balanced audio certainly does not take kindling to L- or R- being connected to ground, or to each other, or both.

   Unbalanced/single-ended simply means that the negative lead is ground, and the positive lead is the audio signal, which is referenced to ground. Most amplifiers expect a single ended signal, and so if you have two channels - like the left and right channels in stereo audio, then the left and right negative terminals are just ground, and will be shorted to ground (and thus, each other) when connected to the amp.

   Balanced/differential signals do not have a ground. Theoretically, they will work without the amplifier and signal source even sharing a common ground (though this is not adviced for EMI/EMC reasons and thus signal integrity). Differential signals are referenced to each other, rather than ground. As an example, a single-ended audio signal might simply be ground on the negative lead, and a wave that swings 1.5V above and -1.5V below that ground reference. In a differential audio signal, the + and - lines are always the same voltage, but opposite polarities. The same signal would result in 3V across + and -, and they would both flip polarity where the single ended signal would show -1.5V.

   Here, let's look at this visually. This is a single-ended/unbalanced signal:

   And the same signal, but as a differential/balanced one:

   

   Your amplifier defaults to single ended/unbalanced signals, and connecting the fully differential analog outs of the RN52 to it is going to result in destroying the noise rejection of one or both channels, depending on the exact architecture of the amp and the analog outputs driving it. It will also simply not work correctly in all sorts of weird and exciting ways.

   If you look at the example circuit in the RN52's datasheet, a good deal of circuitry is needed just to convert the differential audio outputs into something that can drive headphones (the TPA6112 is two op amps in one package, one per channel):

   Anyway, if you unplug the RN52, then of course there is no hum - nothing is connected to the amplifier. If you connect the RN52, you've just connected the ground of the input to something that isn't ground. This is true regardless of whether or not the RN52 is powered up - when unpowered, that pin is still connected to the circuitry inside, and who knows what is coupling into it. Regardless, it isn't ground, and for whatever reason, the effects are only making it onto the left channel. Which channel is seeing the hum, or even the nature of the noise, is not that important.

   Regardless of the hum, you cannot use the 3.5mm jack with the RN52 at all, you must use the two RCA differential inputs, and remove the jumpers. This is mandatory - the signals that the RN52 outputs are not compatible with single-ended inputs.

   This will at the minimum hugely improve the common mode rejection on both channels (in other words, it will reject noise) and not ground stuff to something other than ground. There is a chance something else is wrong with your setup (its impossible to know for sure with the amount of information you've given us and a single, out of focus photo), but I would be quite surprised if simply using the correct audio input for the audio signals you're working with on your amp didn't remove the hum and fix any other weirdness going on. But, you have to do this anyway, so you might as well.

   Good luck!