Windows Server task manager displays much higher memory use than sum of all processes’ working set size

115982 is a very similar question

Is this a 64bit server - do you have the lock pages in memory local policy enabled?

the most likely reason is that this SQL Server 64bit instance is using “locked pages” as I’ve described in an earlier question. Notice the name on the Task Manager column is called Memory (Private Working Set). Remember we also said that if SQL Server 64bit instances use “locked pages” this memory would not be part of the working set (because remember AWE APIs are used on 64bit to “lock” pages and that memory is not part of the working set). So since the locked pages are not part of the working set, they won’t appear in this column in Task Manager. On the Task Manager for Windows Server 2003, this column is called “Mem Usage” but it also reflects the working set of the process.

Here is an article that explains it in depth

You can also view the counters in SQL

SELECT
    object_name
   ,Counter_name
   ,cntr_value
   ,ROUND(( cntr_value * 8192.0 ) / 1048576, 0) AS cntr_value_MB
FROM
    sys.dm_os_performance_counters
WHERE
    object_Name LIKE '%Buffer Manager%'
    AND RTRIM(counter_name) IN ( 'Free pages', 'Total pages',
                                 'Database pages' ) 
UNION SELECT
    object_name
   ,Counter_name
   ,cntr_value
   ,ROUND(( cntr_value / 1024 ), 0) AS cntr_value_MB
FROM
    sys.dm_os_performance_counters
WHERE
    counter_name IN ( 'Target Server Memory (KB)',
                      'Total Server Memory (KB)' )

Windows memory management is a mammoth topic. Entire books can and have been written on just this topic. The first thing I would do is watch the video series, "Mysteries of Windows Memory Management Revealed" parts 1 and 2, on MSDN Channel 9:

http://channel9.msdn.com/Events/TechEd/Europe/2010/WCL401

http://channel9.msdn.com/Events/TechEd/Europe/2010/WCL402

(There are North American and European versions of this session, as he gave these talks at both TechEd in America and again at TechEd Europe.)

Task Manager and Resource Monitor are OK, and are enough for 99% of Windows users, but Sysinternals tools are really stellar when it comes to those of us who are in the last 1% and want to see as much detail as possible.

Start with RAMMap. You can get it at \\live.sysinternals.com\Tools\ . Then to get a better zoomed-in look at how a specific process is using memory, use VMMap.

Let's not go in to trying to define what all those different kinds of memory mean right now. Your problem is that you fear a memory leak. Your symptom is that your server becomes unusable for lack of memory and you're forced to reboot it.

You should never need to reboot a server on a regular basis just from normal operation. Ideally, it should be a self-maintaining system that only needs to be rebooted from patching and configuration changes. So yes, I would say you probably have a problem, assuming you are rebooting your server for a good reason, such as it actually becomes unstable/unusable. Not just because Task Manager is telling you that almost all of your RAM is in use. Modern versions of Windows aggressively cache things and map files and data in RAM, even if you aren't using them right that minute. Note the difference between "Free" memory and "Available" memory in Task Manager.

However, I'm not sure what you mean by "It might be a memory leak in one of the appliances." What do you mean appliance? I thought we were talking about a Windows server here?

You may have a memory leak in several different kinds of memory, such as a processes' private memory, or shared memory, or maybe even pool memory - paged or non-paged. In any of those cases, use those tools that I mentioned and watch those different memory categories for processes that keep rising in memory usage and never come back down. That would be a leak.

Hope this helps.