I finally found the answer to my problem. It's origin are within the application code (like 99% of a programmer's 3rd party tools 'bugs'). I decided to post it anyway in case someone is in a similar scenario.
This code was part of WebServiceRequester class. The web service requester class was instanciated when session was created and it is saved in session. During creation, we initalizate the member 'm_webServiceURL', and this member is saved in session after. At which value was this member initialize was depending on a setting on the local machine.
The important part is the following:
WebServiceRequester class contains a WebService objects.
WebService objects can't be saved in session, they are not serializable in asp. The property had the [NonSerialized] attribute on it. So everytime we accessed the 'WebService' property of the object for the first during a page life cycle, we had to create a new one, and assigning ot it the url 'm_webServiceURL' which was saved in session.
So you see, new webservice object, on possibly a different machine, meaning a different setting on each machine.
so here's what happened:
box 29 was set to access Web Service at localhost
box 30 was set to access Web Service as 192.168.253.29.
Technically, they are both set on the same machine. But here's a scenario:
login on box 29. m_webServiceURL is set to localhost in session.
[some request on box 29 here]
NLB balancing bring us on box 30.
box 30 loads it's session, create a new webservice obect with localhost as the web service address.
box 30 made the request to the wrong web service leading to a Session Expired exception.
One of the problem during the debug, was that the local communication were not recorded with the network monitor.
What lead me on the trace, was that we never had an exception logged on the box 29 log trace, as it should have.
Thanks for you suggestions everyone, it was really appreciated.
Have a good day.
JF
Best Answer
State Server is faster because it stores session data in an in-memory dictionary. SQL Server is slower because it's stored in a database which persists data to disk.
SQL server is also slower because everything is stored in one table which leads to contention as more and more clients access/update the session data.
SQL server is more reliable because it is persisted to disk and can be set up as a cluster with failover capability.
See the preamble in this article for an indepth explanation.