Electrical – Basic Op Amp Question

This should work, but usually there's also a resistor from the non-inverting input to ground:

If \$\dfrac{R4}{R2} = \dfrac{R3}{R1}\$ then:

\$V_{OUT} = \dfrac{R_{3}}{R_{1}} \times (V_{IN+} - V_{IN-})\$

To minimize offset error both inputs have to see the same impedance, and therefore \$R1 = R2\$ and \$R3 = R4\$.
Omitting R4 will only give a scaling factor for the voltage on the non-inverting input, namely:

\$V_{OUT} = 2 \times V_{IN+} - V_{IN-}\$

but changing the setting for the lower potmeter should definitely have a result in the output. Did you measure the voltages on both inputs?

What happens if you set the lower voltage to 2.5V and the upper to 1V? The inverting input should also be 2.5V, and the output 4V. What do you measure?

Note: especially the lower voltage follower is not necessary in your version; the opamp's input current is low enough to be negligible for most uses, and by the way, you're connecting the potmeter's wiper to an exactly same input!

Further reading
differential amplifier tutorial (interesting site overall!)

I suspect the following note from the datasheet may describe what you are seeing:

1. The ALD1722/ALD1722G has complementary p-channel and nchannel input differential stages connected in parallel to accomplish rail to rail input common mode voltage range. This means that with the ranges of common mode input voltage close to the power supplies, one of the two differential stages is switched off internally. To maintain compatibility with other operational amplifiers, this switching point has been selected to be about 1.5V above the negative supply voltage. Since offset voltage trimming on the ALD1722/ALD1722G is made when the input voltage is symmetrical to the supply voltages, this internal switching does not affect a large variety of applications such as an inverting amplifier or non-inverting amplifier with a gain larger than 2.5 (5V operation), where the common mode voltage does not make excursions below this switching point. The user should however, be aware that this switching does take place if the operational amplifier is connected as a unity gain buffer and should make provision in his design to allow for input offset voltage variations

Note the last paragraph. Below 1.5V the offset voltage will vary due to one of the differential stages switching off.
I would try either running from a +/-2.5V supply so the input is at the centre of common mode range, or biasing the input to 2.5V.
Also make sure you are not loading the output too much, e.g. < 5k or so (I assumed not)