First of all, "saturation" in mosfets means that change in VDS will not produce significant change in the Id (drain current). You can think about MOSFET in saturation as a current source. That is regardless of the voltage across VDS (with limits of course) the current through the device will be (almost) constant.
Now going back to the question:
According to wikipedia, the MOSFET is in saturation when V(GS) > V(TH) and V(DS) > V(GS) - V(TH).
That is correct.
If I slowly increase the gate voltage starting from 0, the MOSFET remains off. The LED starts conducting a small amount of current when the gate voltage is around 2.5V or so.
You increased The Vgs above Vth of the NMOS so the channel was formed and device started to conduct.
The brightness stops increasing when the gate voltage reaches around 4V. There is no change in the brightness of the LED when the gate voltage is greater then 4V. Even if I increase the voltage rapidly from 4 to 12, the brightness of the LED remains unchanged.
You increased the Vgs making the device conducting more current. At Vgs = 4V the thing that is limiting amount of current is no longer transistor but resistor that you have in series with transistor.
I also monitor the Drain to Source voltage while I'm increasing the gate voltage. The drain to source voltage drops from 12V to close to 0V when the gate voltage is 4V or so. This is easy to understand: since R1 and R(DS) form a voltage divider and R1 is much larger than R(DS), most of the voltage is dropped on R1. In my measurements, around 10V is being dropped on R1 and the rest on the red LED (2V).
Everything looks in order here.
However, since V(DS) is now approximately 0, the condition V(DS) > V(GS) - V(TH) is not satisfied, is the MOSFET not in saturation?
No it is not. It is in linear or triode region. It behaves as resistor in that region. That is increasing Vds will increase Id.
If this is the case, how would one design a circuit in which the MOSFET is in saturation?
You already have. You just to need take care for operating point (make sure that conditions that you have mention are met).
A) In linear region you can observe following: -> when increasing the SUPPLY voltage, the LED will get brighter as the current across resistor and transistor will rise and thus more will be flowing through the LED.
B) In saturation region something different will happen -> when increasing SUPPLY voltage, the LED brightness will not change. The extra voltage that you apply on the SUPPLY will not translate to bigger current. Instead it will be across MOSFET, so the DRAIN volage will rise together with supply voltage (so increase supply by 2V will mean increasing drain volage by almost 2V)
The current can still flow through the "substrate" even though the channel is pinched. The reason why it saturates is that there will be a region of higher resistance of size proportional to the Drain-Source voltage, and therefore the resistance of this region will be proportional to the same voltage.
But as current is voltage/resistance, the dependence will cancel out and you'll get "constant" current.
From Wiki (emphasis mine):
Even though the conductive channel formed by gate-to-source voltage no longer connects source to drain during saturation mode, carriers are not blocked from flowing. Considering again an n-channel enhancement-mode device, a depletion region exists in the p-type body, surrounding the conductive channel and drain and source regions. The electrons which comprise the channel are free to move out of the channel through the depletion region if attracted to the drain by drain-to-source voltage. The depletion region is free of carriers and has a resistance similar to silicon. Any increase of the drain-to-source voltage will increase the distance from drain to the pinch-off point, increasing the resistance of the depletion region in proportion to the drain-to-source voltage applied. This proportional change causes the drain-to-source current to remain relatively fixed, independent of changes to the drain-to-source voltage, quite unlike its ohmic behavior in the linear mode of operation. Thus, in saturation mode, the FET behaves as a constant-current source rather than as a resistor, and can effectively be used as a voltage amplifier. In this case, the gate-to-source voltage determines the level of constant current through the channel.
Also, from the MOSFET operation description, under saturation:
Since the drain voltage is higher than the source voltage, the electrons spread out, and conduction is not through a narrow channel but through a broader, two- or three-dimensional current distribution extending away from the interface and deeper in the substrate. The onset of this region is also known as pinch-off to indicate the lack of channel region near the drain. Although the channel does not extend the full length of the device, the electric field between the drain and the channel is very high, and conduction continues.
Best Answer
The load line in your graph shows what current flows (Id) over a varying Vgs.
Your text describing triode mode and saturation concerns the behavior of Id over varying Vds (not Vgs) while keeping Vgs constant.
Here the load line crosses the horizontal (well almost) part of the blue curves, that is the saturation region.
The triode mode region is the part on the left where the blue lines are nearly vertical.