W3wp/asp.net High memory usage – Is this problematic

We have a asp.net (.net 4.0) webapplication that is installed in several environments. In most environments, the memory usage is somewhere around 1GB. However, we have one environment where the memory usage peaks to 5.5GB. This is on a Server 2008 machine with 4 cores and 8GB of ram running as an VMWare esx client.

I've set up performance counters with the following results:

Memory
    Committed Bytes    10 145 739 948,0000
    Pages Output/sec                0,0000

Paging File                         _Total
    % Usage                         28,998

Process                             _Total                w3wp
    Working Set              7 480 003 280       5 604 421 056

I also took a memory dump of the w3wp process (when it was +/-2GB, because larger dumps failed). Running DebugDiag on the dump didn't make me any wiser. It seems like .net itself is only taking up 800MB and the bulk of the memory is taken up by 'something else'.

.NET GC Heap Information
GC Heap Size          826,09 MBytes  
Total Commit Size       1217 MB 
Total Reserved Size    16190 MB 

Heap Analysis
Summary
Number of heaps   29 Heaps 
Total reserved memory   1,89 GBytes 
Total committed memory   1,79 GBytes 

...
(largest of the Heaps)
Reserved memory                     1,69 GBytes 
Committed memory                    1,67 GBytes(99,14% of reserved) 
Uncommitted memory                  14,86 MBytes(0,86% of reserved) 
Number of heap segments             113 segments 
Number of uncommitted ranges        113 range(s) 
Size of largest uncommitted range   0 Bytes 

The thing is that I'm not sure that this high memory usage is a problem. So what I'm looking for is some guidance of how to proceed with this issue:

• Either someone tells me this is just how IIS7 works and I shouldn't worry about the memory.
• Or someone points me how I can analyse this dump further (especially how I can see what's in that 1,6GB heap.
• Or explain to me why there is such a big difference between what .net is using and what W3WP is using.

EDIT:
This is what I see in ProcExp:

As you can see, the total Bytes in all heaps is 1.12GB. At the time, the W3WP was using 6.4GB. Why is there such a big difference between these two numbers? What could be taking up this space? Is this the fragmenting of the LOH that I see?