Electronic – LTC3112 undefined behaviour when power supply reaches current limit

Well done!. You have identified a real world effect that many people are unaware of and that can cause significant problems. As well as the undesirable effect of the current spikes on capacitor ripple current (causing reduced lifetime) and possible transformer heating, the sudden current peaks cause RFI (Radio Frequency Interference) from the diodes. Power supplies with too low a primary DC resistance can be RF noise sources! Who would have thought ? :-).

The solution is easy but counterintuitive.
You need "spreading resistance" to increase the capacitor charging resistance. This adversely affects supply "regulation", as Vout drops with increasing load due to the resistor, but a compromise resistor value greatly improves results without too much effect on the regulation. If you are supplying an electronic regulator with this circuit then the drop caused by the resistor can be designed to have no effect at all on Vout, as it simply reduces regulator "headroom" slightly.

Add a small series resistor in the bridge to capacitor lead.
When the bridge voltage rises above Vcap, the higher the current would have been, the greater the drop in the resistor which raises the apparent value of the capacitor voltage temporarily.

Assuming you have about 15VDC peak and want 12 VDC out you have 2V or 3 V headroom. If Ipeak_usual is 300 mA then to drop say 0.5V in the resistor at 300 mA it would have a value of R=V/I = 0.5V/0.3A = 1.66 Ohms. Say 2 Ohms for convenience.

Clearly the prior 2.5A peak is now impossible.
At 2.5A the 2 Ohm resistor would drop V=IR = 2.5 x 2 = 5V and once the capacitor is at working voltage, there is not 5V of "headroom" to drive it.
By changing the R value and looking at the resultant current waveform you can choose a value which is large enough to "spread the conduction angle of the diodes, and small enough to limit the voltage drop as load current increases.

Long long ago undergraduates were given this problem to solve with capacitor value and desired peak diode current as parameters. Without 'modern aids' it was a nasty iterative problem and a good introduction to the real world. Ask me how I know :-).

The difference is that the battery has a higher impedance, especially at the frequencies you are using, than the nicely regulated power supply.

Somewhere in your circuit, probably in the early stages of the audio amplifier, it is making the assumption that ground and power are equivalent for AC signals. From this you can derive that the impedance of the power supply is assumed to be 0. Of course no supply has exactly 0 impedance, but the regulated power supply is a lot closer to this than the battery, and that difference is enough to upset your circuit.

To fix this and to verify this is really what is happening, put a decent size cap across the battery. Try something like a few 100 µF at least, but a mF or more would be better. It would also be a good idea to put a 1 µF ceramic cap across the power feed right where it enters the circuit. That will lower the impedance at high frequencies, something the large electrolytic cap can't do.

If a cap across the battery fixes this, and I think it will, then your circuit was poorly designed. Caps on the power supply should have been included as part of the circuit in the first place, so that it doesn't rely on the impedance of some external source.