Recycling
Recycling is usually* where IIS starts up a new process as a container for your application, and then gives the old one up to ShutdownTimeLimit to go away of its own volition before it's killed.
*- usually: see DisallowOverlappingRotation / "Disable overlapped recycle" setting
It is destructive, in that the original process and all its state information are discarded. Using out-of-process session state (eg, State Server or a database, or even a cookie if your state is tiny) can allow you to work around this.
But it is by default overlapped - meaning the duration of an outage is minimized because the new process starts and is hooked up to the request queue, before the old one is told "you have [ShutdownTimeLimit] seconds to go away. Please comply."
Settings
To your question: all the settings on that page control recycling in some way. "Shutdown" might be described as "proactive recycling" - where the process itself decides it's time to go, and exits in an orderly manner.
Reactive recycling is where WAS detects a problem and shoots the process (after establishing a suitable replacement W3WP).
Now, here's some stuff that can cause recycling of one form or another:
- an ISAPI deciding it's unhealthy
- any module crashing
- idle timeout
- cpu limiting
- adjusting App Pool properties
- as your mum may have screamed at one point: "Stop picking at it, or it'll never get better!"
- "ping" failure * not actually pinging per se, cos it uses a named pipe - more "life detection"
- all of the settings in the screenshot above
What To Do:
Generally:
Disable Idle timeouts. 20 minutes of inactivity = boom! Old process gone! New process on the next incoming request. Set that to zero.
Disable Regular time interval - the 29 hour default has been described as "insane", "annoying" and "clever" by various parties. Actually, only two of those are true.
Optionally Turn on DisallowRotationOnConfigChange (above, Disable Reycling for configuration changes) if you just can't stop playing with it - this allows you to change any app pool setting without it instantly signaling to the worker processes that it needs to be killed. You need to manually recycle the App Pool to get the settings to take effect, which lets you pre-set settings and then use a change window to apply them via your recycle process.
As a general principle, leave pinging enabled. That's your safety net. I've seen people turn it off, and then the site hangs indefinitely sometimes, leading to panic... so if the settings are too aggressive for your apparently-very-very-slow-to-respond app, back them off a bit and see what you get, rather than turning it off. (Unless you've got auto-crash-mode dumping set up for hung W3WPs through your own monitoring process)
That's enough to cause a well-behaved process to live forever. If it dies, sure, it'll be replaced. If it hangs, pinging should pick that up and a new one should start within 2 minutes (by default; worst-case calc should be: up to ping frequency + ping timeout + startup time limit before requests start working again).
CPU limiting isn't normally interesting, because by default it's turned off, and it's also configured to do nothing anyway; if it were configured to kill the process, sure, that'd be a recycling trigger. Leave it off. Note for IIS 8.x, CPU Throttling becomes an option too.
An (IIS) AppPool isn't a (.Net) AppDomain (but may contain one/some)
But... then we get into .Net land, and AppDomain recycling, which can also cause a loss of state. (See: https://blogs.msdn.microsoft.com/tess/2006/08/02/asp-net-case-study-lost-session-variables-and-appdomain-recycles/)
Short version, you do that by touching a web.config file in your content folder (again with the picking!), or by creating a folder in that folder, or an ASPX file, or.. other things... and that's about as destructive as an App Pool recycle, minus the native-code startup costs (it's purely a managed code (.Net) concept, so only managed code initialization stuff happens here).
Antivirus can also trigger this as it scans web.config files, causing a change notification, causing....
A similar thing to that article could be a cause.
The error message is basically "access denied" (anything ending in 05 is typically "no"), which suggests the process tried to do something when it was told to go away, and failed.
Because:
- it's an idle worker process
- which has been told to recycle and
- is thus terminating anyway, and
- a new worker process gets queued up before the old one is terminated (it won't actually be running at this point unless there's a pending request to the same pool; the HTTP.SYS queue gets re-set to WAS in the meantime)
...there should be minimal impact on the app (and zero on any users), unless it's trying to do something crazy on appdomain termination, or similar.
If it was doing it as the App Pool account (discussion of terms), which is really likely (given it's recycling because of no requests) that account presumably failed to read or write something somewhere. Nice and specific isn't it?
So, more information please:
Are you running the App Pool as a specific user account, or as an ApplicationPoolIdentity?
If a specific account, do you have manualGroupMembership=true
- (and if so, is the user a member of IIS_IUSRS? If not using mGM=true, it doesn't need to be, the SID for IIS_IUSRS gets injected anyway)
Best Answer
My guess is that you are hitting the limit of available threads that IIS has available. IIS allocates a certain number of threads depending on the version of IIS and the configuration settings which by default depend on the number of processors/cores in your server. You can tune this specifically to your needs.
You probably run out of threads because you stream downloads to the clients. If you do this in a synchronous way your website will eventually run out of the threads as each thread will be occupied by a single download. If possible you should modify your download code to work asynchronously, e.g. by implementing it as a asynchronous HTTP handler.
EDIT: The 503 error is occuring because IIS is running out of threads to service additional requests and thus can't handle the request. The errors in the event log are a consequence of this as IIS is trying to free up threads and resources by trying to quit/recycle the application pool as it's not responding to the internal 'alive' pings.
Limiting the download speed will actually increase the problem as you will run out of threads even sooner. You should only limit the download speed if you run out of total bandwidth, otherwise just let clients download as fast as they can.
The only real solution that will make your server scale is to change your code so that downloads are handled asynchronously. Or change your solution so that the download is handled by the static file handler of IIS itself, for example by using rewrite rules or a custom rewrite provider. Difficult to say anymore without knowing a bit more details about your application.