Windows memory management is a mammoth topic. Entire books can and have been written on just this topic. The first thing I would do is watch the video series, "Mysteries of Windows Memory Management Revealed" parts 1 and 2, on MSDN Channel 9:
http://channel9.msdn.com/Events/TechEd/Europe/2010/WCL401
http://channel9.msdn.com/Events/TechEd/Europe/2010/WCL402
(There are North American and European versions of this session, as he gave these talks at both TechEd in America and again at TechEd Europe.)
Task Manager and Resource Monitor are OK, and are enough for 99% of Windows users, but Sysinternals tools are really stellar when it comes to those of us who are in the last 1% and want to see as much detail as possible.
Start with RAMMap. You can get it at \\live.sysinternals.com\Tools\ . Then to get a better zoomed-in look at how a specific process is using memory, use VMMap.
Let's not go in to trying to define what all those different kinds of memory mean right now. Your problem is that you fear a memory leak. Your symptom is that your server becomes unusable for lack of memory and you're forced to reboot it.
You should never need to reboot a server on a regular basis just from normal operation. Ideally, it should be a self-maintaining system that only needs to be rebooted from patching and configuration changes. So yes, I would say you probably have a problem, assuming you are rebooting your server for a good reason, such as it actually becomes unstable/unusable. Not just because Task Manager is telling you that almost all of your RAM is in use. Modern versions of Windows aggressively cache things and map files and data in RAM, even if you aren't using them right that minute. Note the difference between "Free" memory and "Available" memory in Task Manager.
However, I'm not sure what you mean by "It might be a memory leak in one of the appliances." What do you mean appliance? I thought we were talking about a Windows server here?
You may have a memory leak in several different kinds of memory, such as a processes' private memory, or shared memory, or maybe even pool memory - paged or non-paged. In any of those cases, use those tools that I mentioned and watch those different memory categories for processes that keep rising in memory usage and never come back down. That would be a leak.
Hope this helps.
I'm not sure what is causing that message, but I know you can create a dump using SysInternals' Process Explorer. While in Process Explorer, right-click on w3wp.exe and select Create Dump > Create Full Dump. It took about 15 seconds on my Windows 2003 x64 server to create a 350 MB dump file. This also did not restart the application pool on my server.
Another option is the Debug Diag tool from Microsoft. This is the ultimate in capturing process dumps, based on many different criteria.
https://www.microsoft.com/downloads/en/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en
Best Answer
This TechNet article should help to explain:
The memory showing in task manager is the amount of physical memory in use by the process that cannot be shared with other processes.
This MSDN page should help explain Virtual Address Space: