Windows memory management is a mammoth topic. Entire books can and have been written on just this topic. The first thing I would do is watch the video series, "Mysteries of Windows Memory Management Revealed" parts 1 and 2, on MSDN Channel 9:
http://channel9.msdn.com/Events/TechEd/Europe/2010/WCL401
http://channel9.msdn.com/Events/TechEd/Europe/2010/WCL402
(There are North American and European versions of this session, as he gave these talks at both TechEd in America and again at TechEd Europe.)
Task Manager and Resource Monitor are OK, and are enough for 99% of Windows users, but Sysinternals tools are really stellar when it comes to those of us who are in the last 1% and want to see as much detail as possible.
Start with RAMMap. You can get it at \\live.sysinternals.com\Tools\ . Then to get a better zoomed-in look at how a specific process is using memory, use VMMap.
Let's not go in to trying to define what all those different kinds of memory mean right now. Your problem is that you fear a memory leak. Your symptom is that your server becomes unusable for lack of memory and you're forced to reboot it.
You should never need to reboot a server on a regular basis just from normal operation. Ideally, it should be a self-maintaining system that only needs to be rebooted from patching and configuration changes. So yes, I would say you probably have a problem, assuming you are rebooting your server for a good reason, such as it actually becomes unstable/unusable. Not just because Task Manager is telling you that almost all of your RAM is in use. Modern versions of Windows aggressively cache things and map files and data in RAM, even if you aren't using them right that minute. Note the difference between "Free" memory and "Available" memory in Task Manager.
However, I'm not sure what you mean by "It might be a memory leak in one of the appliances." What do you mean appliance? I thought we were talking about a Windows server here?
You may have a memory leak in several different kinds of memory, such as a processes' private memory, or shared memory, or maybe even pool memory - paged or non-paged. In any of those cases, use those tools that I mentioned and watch those different memory categories for processes that keep rising in memory usage and never come back down. That would be a leak.
Hope this helps.
Best Answer
Mmmhhhh.
This is a bit of (educated) guesswork based upon similar behavior I have seen with IIS apps.
I think the first time the application is launched by IIS it is called by the same name as the EXE file.
For some reason IIS spawns a seconds copy of the process. This gets the name~1.exe name. A 3rd copy gets name˜2.exe and so on.
It could be that the reason why the extra process spawns is because the first process is no longer responding.
Could also be that the 1st one hangs because of the fact that a 2nd one is launched. (Maybe both processes try to access the same resources causing a deadlock ?)
In my experience such a thing is frequently the result of a misconfiguration of the IIS application pool and/or worker threads.
Anyway: A sort of quick and dirty hack comes to mind: Just periodically check if name˜1.exe is in memory. If it there: Houston we've got a problem.
Then just kill ˜9.exe, ˜8.exe .... ˜1.exe and name.exe whether they are there or not.