Electrical – Op-Amp and MOSFET current source not behaving as expected

Even if the graphs in the datasheet say 1 A at 5 V you should at a load to limit the current. FET characteristics can vary widely, just look at \$V_{GS(th)}\$: minimum 2V, maximum 4 V. For some FETs even a 3:1 range is given. Your 5 V may be 1 V above \$V_{GS(th)}\$, but it might as well be 3 V over it. The 1 A is typical, but \$R_{DS}\$ may already be low. Note that at 10 V this will be as low as 0.5 Ω. Now you have the full 12 V across the FET, and if \$R_{DS}\$ would go to even 2 Ω this would cause a dissipation of 72 W, which without any cooling is way too much for the poor thing.

BTW, even the 12 W of a 1 A current would be too much without cooling, though I wouldn't expect it to explode then.

So make sure you add a load so that the FET won't see the full 12 V across it.

Also, it is important to note that the value of the transconductance, G, specified in the data sheet is NOT the same as the K parameter for a mosfet. They are related by the following expression:

G = 2 * Sqrt( K * Id )

Personally, I would not try to search for the component values that will render that circuit stable. You will have an easier time in the long run configuring the LT1907 as an error amplifier with a cross-over frequency of a few 100 Hz to start.

Here is a basic overview of the 3 classes of error amplifiers to choose from. Source: https://www.powerelectronics.com/power-management/introduction-control-algorithms-switching-regulators

1) I would remove C3 and C6 from your schematic.

2) I would configure the LT1907 as a Type 1 error amplifier.

3) Leave R8 unchanged.

4) Select C1 of error amplifier for a cross-over frequency of 1 kHz.

5) Increase R3 to 1 \$k\Omega\$ for isolation of the amplifier's output from the capactive load of the MOSFET.