This answer is a bit speculative, but its my best guess what's going on.
To get 5 V output, you have your pot turned to get about 720 ohms for "R2". It's got (5-1.25 =) 3.75 V across it, so it's burning about 20 mW. Even though you don't feel it to the touch, it is heating up inside. A pot is probably a relatively large part, which helps explain the very large time constant you're seeing on this voltage drift. Check the datasheet to see if the pot has a postive tempco, or hit it with a hot air gun and watch the output as a first step to confirm or reject this as the source of error.
Another slightly less likely effect, also involving the potentiometer, would be if thermal effects are causing the wiper resistance to drift, or if a chemical change (like oxidation) is happening at the contact of the wiper to the resistive element over time. Since the wiper is part of the "R2" controlling your output voltage, any effect on the wiper resistance will also cause your output to drift.
Depending what kind of pot you're using, and if you really need adjustability, you could try changing to a higher quality pot with lower tempco and known wiper resistance stability, or switching to a fixed resistor, and see if the drift is reduced or eliminated.
Since I cannot easily post images in comments... "there's your problem" as the Hyneman would say.
The voltage drop across Rs is only guaranteed if certain conditions are met:
- minimum voltage drop from input to output [of LM317's pints] of 3V, and
- minimum load current of 10mA, and
- power dissipation below 20W.
In this case the first condition wasn't met.
Best Answer
You're loading the 1.25V output of the LM317 with a transistor- it won't work very well. It will burn the pot out if you turn it up all the way.
I assume you're trying to draw a constant current from V1.
One method is to use an op-amp such as 1/2 an LM358:
simulate this circuit – Schematic created using CircuitLab
With the values shown, it will allow a current from 0-1A to be drawn from V1. R1 will dissipate up to 1W, and Q2 will dissipate most of the rest of the power- up to 2.7W for the values shown.
R2 is a voltage divider that provides an adjustable 0-1V signal on the non-inverting input. R4 provides a bit higher maximum output voltage from OA1. Q1 and Q2 form a darlington pair. R1 is the current sense resistor.