Win 2012 R2 / IIS 8.5 intermittent Connection Refused

iis-8.5networkingwindows-server-2012-r2

We suffer from a connection refused problem when the users of our web site try to open it. This problem happens in a random manner, about once or twice a month, and problem continues for a few hours. Also when happening, almost all connections are rejected by connection refused error. but there are successful connections meanwhile.

  • OS: Win 2012 R2 Standard hosted on ESXI 6
  • IIS 8.5
  • Web server is hosting an ASP.NET application.
  • Windows Firewall is on.
  • Average current connection on server: ~3500 (based on Web Service\Current connection performance monitor counter)
  • Total RAM: 40GB
  • CPU: 30 cores, 2.30 GHz

There is plenty of RAM (more than ~60%) and CPU (more than ~70%) available while this problem happens. Also we checked the network firewall and apparently traffic is passing through network firewall without problem and problem happens at the server level. And we can not even open the web site by doing Remote desktop and trying to open in locally.

We checked about exhausted port problem and it seems that is not the problem.
The number of SYN packets are high, but its similar to other days when everything is fine.

This is one day summery of HTTPERR log:

s-reason    COUNT(ALL *)
Timer_ConnectionIdle    462040
Timer_MinBytesPerSecond 27555
Request_Cancelled   1757
Timer_EntityBody    428
Forbidden   247
URL 130
Hostname    117
BadRequest  102
Connection_Dropped  96
Client_Reset    88
Connection_Dropped_List_Full    40
Verb    10
Header  7
Connection_Abandoned_By_ReqQueue    1

Any help is really appreciated to find the reason why we get connection refused when trying to open web site hosted on this server.

Best Answer

Are you running in a virtual environment, or a physical machine? (Edit, just re-read, and saw ESXI 6. So Virtual it is then.)

You've got a VMware VM, are you using the standard install NIC, or are you using the VM providers specific NIC? (ie: Intel vs. VMWare)

We have a similar issue, but is much less persistent when it occurs. (Ours is exposed when an automated script runs to check 30 sites up/down status, but only effects a half dozen LWP gets.)

(Sorry, can't use comments yet, don't have the rep for it)

Edit 2

As per this TechNet Link Mohammad found, SYN Attack Protect is on by default in >= Vista. Which is what I found yesterday, but unlike what I read yesterday, the RegEdits in Edit 1 apparently don't make it more aggressive or active.

I've taken a temporary approach of blocking IPs at the firewall to see what happens. Excess SYN_RECEIVED connections drop, and then eventually rise again on another IP (As you would expect).

Edit 1 - [Possibly debunked?]

If you haven't got to reading all the comments below, it looks like this is headed in the direction of a SYN Attack (for both of us).

I'm currently trialling the following changes in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\ on a development server for testing:

SynAttackProtect: https://technet.microsoft.com/en-us/library/cc938202.aspx TcpMaxConnectResponseRetransmissions: https://technet.microsoft.com/en-us/library/cc938208.aspx TcpTimedWaitDelay: https://technet.microsoft.com/en-us/library/cc938217.aspx TcpMaxHalfOpenRetried: https://technet.microsoft.com/en-us/library/cc938213.aspx TcpMaxPortsExhausted: https://technet.microsoft.com/en-us/library/cc938214.aspx TcpTimedWaitDelay: https://technet.microsoft.com/en-us/library/cc938217.aspx

Somewhat like what is detailed here: https://alnitech.com/news/how-to-protect-your-windows-server-from-syn-flood/

Useful note - some TCP / UDP Port usages. Especially if you're considering increasing ports in the ephemeral range. (ie: netsh int ipv4 set dynamicportrange tcp start=45536 num=20000) https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers

Related Topic