In IIS, an Application runs within an Application Pool. An Application Pool runs in one or more W3WP processes.
It sounds like what's happening here is that your app eventually encounters a condition that the W3WP can't cope with when run with minimal privileges.
LocalSystem is a high-privileged account - when stuff starts (or in your case, keeps) working as LocalSystem, typically that indicates some form of privilege level difference (occasionally, user profile differences).
All App Pool Identities are magically given membership to IIS_IUSRS by default, which grants all the permissions IIS requires to run an App Pool - this may not include all the permissions required by your particular app/framework/one of the libraries.
Try enabling the Load User Profile setting in the app pool when running low-privileged, and if that doesn't improve things, you'll need to debug it - either watch it running with Process Monitor for possible file or registry access difficulties (http://live.sysinternals.com/procmon.exe), or grab a memory dump of the process when it's failed, grab PSSCOR2, and perhaps start with !dae if nothing looks particularly odd at that point.
The short version is that the internal process / runtime state sounds like it's getting lunched; it shouldn't; often it's going to be some sort of native code issue that does that, but in your case, it could be permissions.
Edit: Oh, not LocalSystem, {local system user like administrator}. Gotcha. Well, give the Profile thing a try - see also this bit.
Also, Recycling: when a new W3WP is started to replace the old one, losing all the state from the old one in the process, and the old one is killed off after 90 seconds if it doesn't complete any useful work it's still capable of, and shut itself down. (read "by default")
Recycling
Recycling is usually* where IIS starts up a new process as a container for your application, and then gives the old one up to ShutdownTimeLimit
to go away of its own volition before it's killed.
*- usually: see DisallowOverlappingRotation
/ "Disable overlapped recycle" setting
It is destructive, in that the original process and all its state information are discarded. Using out-of-process session state (eg, State Server or a database, or even a cookie if your state is tiny) can allow you to work around this.
But it is by default overlapped - meaning the duration of an outage is minimized because the new process starts and is hooked up to the request queue, before the old one is told "you have [ShutdownTimeLimit
] seconds to go away. Please comply."
Settings
To your question: all the settings on that page control recycling in some way. "Shutdown" might be described as "proactive recycling" - where the process itself decides it's time to go, and exits in an orderly manner.
Reactive recycling is where WAS detects a problem and shoots the process (after establishing a suitable replacement W3WP).
Now, here's some stuff that can cause recycling of one form or another:
- an ISAPI deciding it's unhealthy
- any module crashing
- idle timeout
- cpu limiting
- adjusting App Pool properties
- as your mum may have screamed at one point: "Stop picking at it, or it'll never get better!"
- "ping" failure * not actually pinging per se, cos it uses a named pipe - more "life detection"
- all of the settings in the screenshot above
What To Do:
Generally:
Disable Idle timeouts. 20 minutes of inactivity = boom! Old process gone! New process on the next incoming request. Set that to zero.
Disable Regular time interval - the 29 hour default has been described as "insane", "annoying" and "clever" by various parties. Actually, only two of those are true.
Optionally Turn on DisallowRotationOnConfigChange (above, Disable Reycling for configuration changes) if you just can't stop playing with it - this allows you to change any app pool setting without it instantly signaling to the worker processes that it needs to be killed. You need to manually recycle the App Pool to get the settings to take effect, which lets you pre-set settings and then use a change window to apply them via your recycle process.
As a general principle, leave pinging enabled. That's your safety net. I've seen people turn it off, and then the site hangs indefinitely sometimes, leading to panic... so if the settings are too aggressive for your apparently-very-very-slow-to-respond app, back them off a bit and see what you get, rather than turning it off. (Unless you've got auto-crash-mode dumping set up for hung W3WPs through your own monitoring process)
That's enough to cause a well-behaved process to live forever. If it dies, sure, it'll be replaced. If it hangs, pinging should pick that up and a new one should start within 2 minutes (by default; worst-case calc should be: up to ping frequency + ping timeout + startup time limit before requests start working again).
CPU limiting isn't normally interesting, because by default it's turned off, and it's also configured to do nothing anyway; if it were configured to kill the process, sure, that'd be a recycling trigger. Leave it off. Note for IIS 8.x, CPU Throttling becomes an option too.
An (IIS) AppPool isn't a (.Net) AppDomain (but may contain one/some)
But... then we get into .Net land, and AppDomain recycling, which can also cause a loss of state. (See: https://blogs.msdn.microsoft.com/tess/2006/08/02/asp-net-case-study-lost-session-variables-and-appdomain-recycles/)
Short version, you do that by touching a web.config file in your content folder (again with the picking!), or by creating a folder in that folder, or an ASPX file, or.. other things... and that's about as destructive as an App Pool recycle, minus the native-code startup costs (it's purely a managed code (.Net) concept, so only managed code initialization stuff happens here).
Antivirus can also trigger this as it scans web.config files, causing a change notification, causing....
Best Answer
It sounds like something could be binding to the port. Trying using netstat to see what is listening on port 80.
The following command will list each listening process, the port and the process ID. You can then use the process ID to determine the what is using that port.