TDA2030A gain reduction for Headphone Amp

Alfred already explained that for a voltage source (which an amplifier is) you'll have less power at higher impedance, because the current decreases. If your amplifier would be a current amplifier you would get a higher power, because the same current in a higher impedance will increase voltage.

Citing from this document, because I can't explain it better myself:

Headphone manufacturers specify a “sensitivity” rating for their products that is very similar to loudspeaker sensitivity ratings. For loudspeakers, the standard is to apply 1 watt and then measure the sound pressure level (SPL) at a distance of 1 meter. For headphones, the standard is to apply 1 milliwatt (1 mW = 1/1000 of a watt) and then measure the sound pressure level at the earpiece (using a dummy head with built-in microphones). Sensitivity is then stated as the number of dB of actual sound level (SPL) produced by the headphones with 1 mW of input; headphone specifications commonly refer to this by the misleading term “dB/mW.” What they really mean is dB SPL for 1 mW input.

Think about these sensitivity definitions a moment: headphone sensitivity is rated using 1/1000 of a watt; loudspeaker sensitivity is rated using 1 watt. So a quick rule-of-thumb is that you are going to need about 1/1000 as much power to drive your headphones as to drive your loudspeakers since both of their sensitivity ratings are similar (around 90- 110 dB SPL). For example, if your hi-fi amp is rated at 65 watts, then you would need only 65 mW to drive comparable headphones. (Actually you need less than 65 mW since most people don’t listen to their loudspeakers at 1 meter.) And this is exactly what you find in hi-fi receivers—their headphone jacks typically provide only 10-20 mW of output power.

Take another moment and think about all those portable tape players. They sound great, and loud. Why, you can even hear them ten feet away as the teenage skateboarder that ran over your foot escapes.

Power output? About 12 mW.

(emphasis by me)

Thanks to marketing numbers of 100 W amplifiers most people don't realize this, but 1 W is a lot of power for a good speaker. It can give you more than 90 dB SPL at 1 m. At full power a 100 W amplifier just won't break the windows. Claiming to play 2000 W at full power in your living room is nothing to brag about: it just says that you have lousy speakers :-). 2000 W in 92 dB speakers delivers 125 dB SPL, which will turn you deaf in no time. (That may be OK, once you're deaf it also stops hurting your ears. :-)

Further reading
Understanding headphone power requirements

First thing I notices is the time constant with your input capacitance and 1Mohm resistor - it's 47 seconds or equivalent to a high pass 3dB frequency of 0.0034 Hz.

This might take ages for your inputs to settle - make the input capacitor 47nF is my recommendation - this puts the high pass frequency at about 3.4Hz. Even a 10nF wouldn't be unreasonable (high pass of 16 Hz).

Your op-amp is a TL084 so I'm less worried about input current leakages causing offset voltage problems AND I'm happier with non-polarized caps at the inputs because if it is an electret mic and has a bias voltage on it, the electrolytic (have to be polarized correctly) input caps will become reverse biased and this may indeed be a possible reason why you see a problem.

Regarding your 50k pots, these will potentially give higher noise than say 5kpots and with 5k pots you can reduce R5, R10, R15 and R16 to something like 10ks also reducing noise. Noise is proportional to the square root of resistance so always best to go lower a bit if you can.

C7 and C12 are also problematic in the the signal they see may become reverse polarized. The time constants you are using around these circuits are very large given that those caps are 47uF - go for 1uF and non-polarized. U2 just appears to be a gain of ten stage so lose the centre biasing - 0V is centre and that is just fine assuming your Vcc and Vdd are largely equal and opposite.

You are using TL084 output amplifiers but this is really daft because you have an output pot that will likely introduce an impedance of about 25k (mid point). Just use pots and get rid of the extra 3 op-amps that feed the individual outputs. C9, C10 and C11 are not needed either or if you feel happier with them use non-polarized 1uF caps.