First of all, "saturation" in mosfets means that change in VDS will not produce significant change in the Id (drain current). You can think about MOSFET in saturation as a current source. That is regardless of the voltage across VDS (with limits of course) the current through the device will be (almost) constant.
Now going back to the question:
According to wikipedia, the MOSFET is in saturation when V(GS) > V(TH) and V(DS) > V(GS) - V(TH).
That is correct.
If I slowly increase the gate voltage starting from 0, the MOSFET remains off. The LED starts conducting a small amount of current when the gate voltage is around 2.5V or so.
You increased The Vgs above Vth of the NMOS so the channel was formed and device started to conduct.
The brightness stops increasing when the gate voltage reaches around 4V. There is no change in the brightness of the LED when the gate voltage is greater then 4V. Even if I increase the voltage rapidly from 4 to 12, the brightness of the LED remains unchanged.
You increased the Vgs making the device conducting more current. At Vgs = 4V the thing that is limiting amount of current is no longer transistor but resistor that you have in series with transistor.
I also monitor the Drain to Source voltage while I'm increasing the gate voltage. The drain to source voltage drops from 12V to close to 0V when the gate voltage is 4V or so. This is easy to understand: since R1 and R(DS) form a voltage divider and R1 is much larger than R(DS), most of the voltage is dropped on R1. In my measurements, around 10V is being dropped on R1 and the rest on the red LED (2V).
Everything looks in order here.
However, since V(DS) is now approximately 0, the condition V(DS) > V(GS) - V(TH) is not satisfied, is the MOSFET not in saturation?
No it is not. It is in linear or triode region. It behaves as resistor in that region. That is increasing Vds will increase Id.
If this is the case, how would one design a circuit in which the MOSFET is in saturation?
You already have. You just to need take care for operating point (make sure that conditions that you have mention are met).
A) In linear region you can observe following: -> when increasing the SUPPLY voltage, the LED will get brighter as the current across resistor and transistor will rise and thus more will be flowing through the LED.
B) In saturation region something different will happen -> when increasing SUPPLY voltage, the LED brightness will not change. The extra voltage that you apply on the SUPPLY will not translate to bigger current. Instead it will be across MOSFET, so the DRAIN volage will rise together with supply voltage (so increase supply by 2V will mean increasing drain volage by almost 2V)
On startup, your bootstrap capacitor C3, C4 need be charged through D2 to VS then R1 to ground. If your R1 is too large, the charging time will be long. Assume R1 = 1k ohms, and C3 = 22uF, then
$$
\tau = C3 \times R1 \approx 22ms
$$
This causes about 50ms 10% ~ 90% rising time. Because the high side driver supply actually floats on your load, so as the capacitor voltage rising, the lower slope line reflect the voltage change on your load resistance.
When you change the load resistor to 10 ohms, the time reduced to about 500us.
Best Answer
From the IRF640N datasheet, on page 3, we have these two graphs:
The red dots show the point on the \$V_{GS}=7V\$ line where current is 2A. Fig. 1 is good for when the transistor is cool, at 25°C, and you can see that it will develop a drain-source voltage of a little over \$V_{DS}=0.2V\$. This corresponds to a power dissipation of:
$$ P_{25} = I \times V = 2A \times 0.2V = 0.4W $$
This will obviously cause the transistor to heat up, quite slowly at first. From fig. 2 it's clear that as temperature rises, so does \$V_{DS}\$. If current is kept constant at 2A, then as \$V_{DS}\$ rises, so does power dissipation, causing the transistor to heat faster. This leads to \$V_{DS}\$ rising faster still, causing even faster heating.
If ever the temperature of the transistor were to rise to 175°C, fig 2. shows that with 2A of drain current \$V_{DS}=0.6V\$, and power would then be:
$$ P_{175} = I \times V = 2A \times 0.6V = 1.2W $$
By then, the transistor would surely be cooking itself very quickly indeed, unless you could remove that heat as quickly as it's generated, with a heatsink.
This problem occurs when drain current is kept constant, and the phenomenon is called thermal runaway. Thermal runaway generally does not occur if the load is a fixed resistance, instead of something that varies resistance to maintain constant current. That's because if the load was resistive, the voltage across it would fall as \$V_{DS}\$ rises, causing current to fall also, preventing the transistor's power dissipation from rising over time.
I think you'll have less destructive results using your programmable load in constant resistance mode instead of constant current.