Electronic – Voltage Follower Circuit Output Changes based on Load Resistance

I could see this being on/off control (a switcher) if there was a inductor in series with the FET. However since there isn't, this won't work as a switcher.

This topology looks like it is intended for linear operation. However, the capacitor slows down the system response so that the controller could easily oscillate under some conditions. This is not a good idea in this case. The way to fix it is to slow down the controller. This can be done by "compensating" the opamp if it has such control. If not, the right size cap between the opamp output and its - input will slow it down. For that to work, there will have to be a resistor between Vout and the - input. You can start with a 1 kΩ to 10 kΩ resistor and then use the capacitor value that is just a little above the minimum where the system doesn't oscillate at all operating points you care about.

There are more sophisticated ways to control power supplies, but what I described above is a quick fix that only adds two parts to the existing topology. The tradeoff is that is will slow the transient response. If you care about performance, then you need to start from the beginning and design a power supply the right way.

After further reading into non-ideal op-amp behavior it looks like it's actually the op-amp input bias current that's the culprit.

R1||R2 ~= 100kohm, which created about a 0.6V drop (close enough to what the difference was). Adding a resistor to the inverting input equivalent to R1||R2 fixed the problem.

Reference: Practical Considerations: Bias current

The fixed circuit: