Electrical – Op-amp gain decrease, output not linear

You are trying to use the "opamp out of the range it behaves well". Used open loop like this it becomes a makeshift comparator.

Opamps are meant to always be used closed loop. The open loop gain is pretty variable so cannot be relied upon, the only real requirement is that it's as large as possible (ideally infinite) When you close the loop though the huge gain makes the (closed loop) gain very accurate and the open loop variability (usually) makes almost no difference.

In theory, with an ideal opamp what you are trying would work okay, but in real opamps you have slight mismatches, causing things like input offset voltage which just by itself will cause this experiment to fail.

If we look at page 3 of the datasheet, we see the typical offset voltage for the LM324 is 2mV, up to a maximum of 7mV (3mV for the LM324A). This swamps your tiny input signal, and will cause the output to do just about anything, probably saturate.
There is also an offset current, external noise, and other non-ideal parameters that are variable with temperature that will cause issues here.
For an example of how variable the open loop gain is, here is a graph of the open loop gain over supply voltage:

We can see at 5V, the open loop gain is around 100dB (100,000) compared to 120dB (1,000,000) at 15V. This is a factor of 10 difference. Also we can see it varies considerably with load.

Most likely a common-mode voltage violation. See Figure 15 of the datasheet. Unfortunately it does not give you the limits for a single power supply, but shows you that common-mode and output voltage limits go hand in hand.

Actions that may solve your problem:

• Use dual power supply.
• Lift inputs above ground.
• Use a lower gain setting.

EDIT:

As pointed out by Dave Tweed, the datasheet does have information on common-mode and output voltage ranges for single supply. See Figure 13 on page 10 of the datasheet (your case is the blue triangular area). Your common-mode input is practically ground, so as soon as your output exceeds about 0.7 V, the limit is exceeded, and your device will not behave as specified. This will happen even if you change Vref to 2.5 V: the output limit will be 3.2 V for a common-mode input voltage of ground to become invalid.