You are assuming the speakers are driven with single phase and ground. That may be a BAD assumption. The speakers may be driven in push-pull mode.
See this cross post.
Try connecting ONLY the L, and R and find a different ground.
This question is pretty old, but I can maybe elaborate on what the other answers are saying.
The socket has five pins - You might think this is odd, because your stereo plug only has three connections: Left, Right, Ground. What are these strange extra connections?
The symbol you have shows two connections with weird angled parts at the end. Those are the bits that actually interface with the plug. The top one is the tip, and the bottom one is the ring. The sleeve is connected to gnd. The other two connections are the ones that act "odd". Each of these is normally connected to one of the angled contacts, except when a plug is inserted. When the plug is inserted, they are not connected to anything.
This means that you can use these connections to detect if a plug is inserted or not. Going even further, you can use these to directly solve your problem. If you connect your audio source to the two angled pins, when nothing is inserted, audio will come out of the two non-angled pins, so you should connect those to your speakers. When the plug is inserted, it will disconnect those two pins, muting the signal to your speakers.
You shouldn't need any of those other components, except the potentiometer, which will need to be far smaller, around 20 ohms as Peter says. If you want to control the speaker and the jack with the same potentiometer, it should be in series with the audio, and you'll likely need two of them, or a dual one (to control both channels). Dont be tempted to put a single pot between the jack and ground, because you'll get strange sounding music as the common signals start to attenuate.
Best Answer
Since you are pretty new to electrical engineering, I will give you a complete circuit, with explanation.
Turning on a relay requires a fair amount of current (~50mA) which is more than typical ICs want to source directly, so usually a transistor is used to take a small control current and switch on the large relay current. This circuit uses any generic PNP transistor, e.g. 2N3906.
The chip shown is any generic “open collector comparator”, e.g. one section of an LM339. These devices pull low (to ground in this case) when their ‘-‘ input is higher than their ‘+’ input. This condition happens when any audio signal shows up that exceed the threshold voltage (set by the 1M / 22K resistor but can be adjusted for your desired trigger point).
When a positive audio peak exceeds the threshold volage then the comparator goes low, discharging the capacitor and turning on the PNP transistor and hence the relay. The capacitor will continually get reset as audio continues to come in.
When audio stops, the comparator’s ‘-‘ input stays at ground and now its ‘+’ input is greater so the comparator goes high. But “high” for an open collector device really means that it disconnects entirely from the output so only the 1M resistor is left to slowly charge the capacitor, eventually all the way to the +V rail at which point the transistor turns off (because its base-emitter voltage is zero). The 1M / 1uF creates a time constant of a second or two that keeps the circuit on after audio goes away. Making C bigger will keep it on even longer.
The +V voltage rail must be chosen to match the relay’s specified voltage. Typical relay voltages are 5V, 9V and 12V. Depending on the +V you end up with you may want to fiddle with the 1M/22K values to get the audio turn on threshold right.
For simplicity you may be tempted to run the whole thing from a 9V battery. That would probably work but not for very long: the ~50mA constant current requirement would drain the battery quickly. Better to use 5V from a USB port or a 5/9/12V DC adaptor.
Lastly, a diode (any diode) is usually placed across the relay coil to give the pent up current in the coil a discharge path when abruptly turned off.