I've been working on it for a few weeks. I've an IIS 8.0 Server hosting, ASP .NET 4.0 application in Classic mode. Two of the files are GET which do file system reads for every request and one is post which writes to file system. Now, I sometimes start getting "Server too Busy" from a server. That error is intermittent when the server reaches this state, i.e., some of the requests return 200 while most of them return "Server Too Busy". And, even restart of IIS server doesn't make it behave normally.
I read regarding the possible causes of this issue. Basically, Queue size exceeding a limit, timeout of requests in queue, etc. should be the reason. Now, there are queues are various levels, HTTP.sys for Application pool, ASP .Net global queue, Application (AppDomain) queue, CLR queue.
I tried to reproduce the error by sending lots of GET requests to the server simultaneously, but none of the queue size seems to be increasing much as per perfmon. I started getting "Connection Timeout" due to this, but not "Server Too Busy". Where am I going wrong in reproducing this? Is it because I am sending the same request every time and it is getting cached by http.sys cache itself?
To make sure, I am able to figure it out the next time I encounter it, I want to log various queue sizes periodically, number of requests pending on the receiver, the number of requests for each of the API currently in process, i.e., log any of the parameters that may cause this problem periodically. How can I do that?
I didn't get any error log in ASP .NET logs. In HTTPERR, I can find many "Timer_ConnectionIdle" errors and very few "503 N/A" (Service Not Available).
Please help. Thanks in advance.
Edit: On Tracing the "Server Busy" request, I got a warning events. Here is the trace of that:
-MODULE_SET_RESPONSE_ERROR_STATUS
ModuleName
IsapiModule
Notification
EXECUTE_REQUEST_HANDLER
HttpStatus
503
HttpReason
Server Too Busy
HttpSubStatus
0
ErrorCode
The operation completed successfully.
(0x0)
Can anybody please provide any insight into it?
Best Answer
In terms of recreating the problem, the way most web servers (including IIS) handles multiple connections from one address/client is different than how it handles connections from multiple addresses/clients. Consider running a load generating tool (particularly one that can spoof multiple concurrent IP addresses or at least multiple virtual clients from the load generator) to get a better view of what might be happening. I haven't used this, but Microsoft apparently recommends a tool for this: http://blogs.msdn.com/b/asiatech/archive/2013/01/21/how-to-perform-a-stress-test-with-wcat-tool.aspx and http://www.iis.net/downloads/community/2007/05/wcat-63-%28x64%29