Iis – Unable to get the private bytes memory limit for the W3WP process

Limit the Private Memory (Committed bytes) this is what corresponds to your actual Memory usage.

Virtual Memory isn't shared across applications and on x64 worker processes will generally reserve much much, more than they actually end up using. Forcing limits on this really only makes sense on x86.

IIS will respect both of those limits.

If you set a 500MB private byte limit, as soon as a worker process attempts to commit 501MB, IIS will spin up a new worker process and kill the old one.

If you set a 3GB virtual memory limit, as soon as a worker process attempts to reserve 3.001GB, IIS will spin up a new worker process and kill the old one.

If you are on a 64bit platform, you should be aware that ASP.NET applications aggressively reserve virtual memory. As an example, I have an app on a farm that uses only 88MB of private bytes, but its sitting at 5.4GB Virtual Size right now. I believe the virtual memory reservation is a function of physical RAM on the server. It's also important to understand that on a 64bit platform, reserving large portions of virtual memory has zero performance impact.

Basically, if you are having memory consumption issues on an IIS server, the setting you want to limit is Private Memory/Bytes, this is what corresponds to actual memory usage.