Electrical – Level shifter using LM324 Op Amp for ADC reading

This should work, but usually there's also a resistor from the non-inverting input to ground:

If \$\dfrac{R4}{R2} = \dfrac{R3}{R1}\$ then:

\$V_{OUT} = \dfrac{R_{3}}{R_{1}} \times (V_{IN+} - V_{IN-})\$

To minimize offset error both inputs have to see the same impedance, and therefore \$R1 = R2\$ and \$R3 = R4\$.
Omitting R4 will only give a scaling factor for the voltage on the non-inverting input, namely:

\$V_{OUT} = 2 \times V_{IN+} - V_{IN-}\$

but changing the setting for the lower potmeter should definitely have a result in the output. Did you measure the voltages on both inputs?

What happens if you set the lower voltage to 2.5V and the upper to 1V? The inverting input should also be 2.5V, and the output 4V. What do you measure?

Note: especially the lower voltage follower is not necessary in your version; the opamp's input current is low enough to be negligible for most uses, and by the way, you're connecting the potmeter's wiper to an exactly same input!

Further reading
differential amplifier tutorial (interesting site overall!)

The few tens of mV noise is almost surely caused by the 7660, which has an operating frequency of around 10kHz.

The PSRR of the TL084 is much worse (14dB) than the LM324.. Have a look at the actual ripple on the negative supply and related layout.