Output voltage of unity-gain buffer varying with current

I'm working on a VLC+IR transmitter that runs from batteries. I decided to use an LM317 so that the discharging battery wouldn't really affect the current through the LEDs. (Disclaimer: I've never worked with this IC before in my life.) I use TINA for simulations and I found a spice model of the 317 somewhere on the internet.* Its output voltage dropped when connecting the LEDs, so I connected a unity-gain buffer op-amp between the two.

The transmitter has two "outputs": a white and an IR LED. They can work together or one-by-one. VG1 and VG2 are square waves with a frequency of 1 MHz. When I turn off SW-SPST2 or SW-SPST3, the other is working properly. However, when I try to operate them at the same time, one of them stops behaving normally. I checked the op-amps output voltage (VM3), and it keeps changing between 2 and 5 volts, approximately. I don't understand why, as the op-amp has a really low output impedance, thus the output voltage shouldn't be affected by the load.

So, my question is: First, what is the cause? Simply a simulation mistake? Second, I'm not really convinced that the LM317 model is correct, so, am I correct assuming, that its output voltage drops with a load?

Edit: Yes, I made a mistake by putting a 2k resistor to the LM317. I just wanted to produce a stable ~4 volts. Now, I replaced it with 540 Ohms, the nominal value for which the output is about 4.06V. The problems are still there, whether or not I use the op-amp. If I don't use it, the voltage drops after I connect the load. If I do use it, I still can't use both the LEDs at the same time.