Nginx – When a server IP changes, do exising TCP (e.g. http/thesql) connections remain running

The issue is caused by your extremely large timeout. With a 24-hours timeout and a limit to 1000 concurrent connections, you can clearly expect to fill this with clients disconnecting the dirty way. Please use a more reasonable timeout, from minutes to hours at most, it really makes no sense to use 1 day timeouts on the internet. As DukeLion said, the system is waiting for haproxy to close the connection, because haproxy did not receive the close from the client.

Haproxy being working in tunnel mode for TCP and WebSocket, it follows the usual 4-way close :

- receive a close on side A
- forward the close on side B
- receive the close on side B
- forward the close on side A

In your case, I suppose that side A was the server and side B the client. So nginx closed after some time, socket went to CLOSE_WAIT, haproxy forwarded the close to the client, this socket went to FIN_WAIT1, the client ACKed, passing the socket to FIN_WAIT2 and then nothing happens because the client has disappeared, which is something very common on the net. And your timeout means you want this to remain this way for 24 hours.

After 24 hours, your sessions will start timing out on the client side so haproxy will kill them and forward the close to the nginx side, getting rid of it too. But clearly you don't want this to happen, WebSocket was designed so that idle connection could be reopened transparently, so there is no reason to keep an idle connection open for 24 hours. No firewall will keep it along the way !

It appears to me that you might be in violation of the license agreement since you are using 2008 Web Edition:

From Here:

Windows Web Server 2008 is specifically designed to be used as a single-purpose Web server . It is intended only for Internet accessible, front- end Web serving of Web pages, Web sites, and Web applications.

Anyway, the Web Edition of Server 2008 does indeed have a limit of 10 simultaneous half-open TCP connections, but no reasonable limit on active TCP connections, according to this Microsoft employee:

It depends on the edition, Web and Foundation editions have connection limits while Standard, Enterprise, and Datacenter do not.

But if you're consistently seeing the number of connections get stuck at exactly 1000, consistently, then I'm not yet convinced that half-open TCP connections are your problem. Couldn't tell you more without scrutinizing your code.

But you should probably move off of Web Edition anyway, since you are using it for something other than running a web server. So the best answer I think you'll get given the information you've provided to us is no, there is no limit of active TCP connections that is anywhere near 1000 that is imposed either by the operating system or by the .NET framework. It is capable of handling many, many more than that.

Edit: I suppose you could try changing the registry key HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\EnableConnectionRateLimiting to 0, but that may or may not work because of your Web Edition of Windows Server, as well as opening you up to possible denial of service attacks (if you actually are having a problem with half-open TCP connections, which I'm not yet convinced that you are. Run a netstat and see if you have a bunch of connections that seem to be stuck in a SYN_SENT state.)