High Memory Utilization on a few servers in an ESXi Environment

A couple of things:

1. Don't use Limits on RAM. To cap the amount of RAM the VM can use, use the amount of RAM when definining the VM. Otherwise, if you set a limit for RAM on a VM smaller than the amount of RAM that's defined for the VM, ESX will be forced to page. If paging is to occur, you want the guest to page inside itself because it's more intelligent about what data should be paged than ESXi is. When definining a VM, you want to define the RAM as the highest amount of RAM the VM might need, or the cap you want the VM to use.

2. Unless absolutely necessary, do not use reservations for any VM on any resource, including memory. If you install VMTools, which you always should unless you have a good reason not to, ESXi can reclaim memory, particularly when it's not being used. Setting reservations will force ESXi to provide physical memory even if the guest doesn't need it. Instead, use Shares to set priority when there is resource contention.

The above is very important. If you do not follow those, what will happen is you'll have inefficient paging, and inefficient granting of resources because VM's that don't need RAM will get it anyway, and VM's that could use the RAM won't get it. You'll also often times have VM's that need RAM, but may not get it even if there is some to provide.

The difference between those two screens comes down to the summary being how much RAM is actually being used, and total capacity shows the amount that could be used adjusted after reservations.

To answer your question, this is a reservation setting for the Hypervisor. This illustrates actually my above point that you shouldn't use reservations unless you abslutely have to. The ESXi hypervisor has perhaps too high of a memory reservation. You can tweak this. Reference the following article:

http://vm-help.com/esx40i/memory_allocation.php

I don't recommend doing that for production though.

You want to use the most advanced network drivers you can for every guest. Windows server 208 r2 supports the paravirtualization drivers. Paravirtulized drivers present the least overhead to both guest and host. If you are forcing the host to completely emulate a piece of hardware, you are forcing CPU utilization. Switch to vmxnet3 and see if that reduces the workload and increases performance. You also mention that you are looking at the guest for performance issue. This is a lie. you should always reference the vmware performacnce counters instead of the guest performance counters as the system is virtualized. as an example you might see the cPU at 100% but in vmware you will realize that it's 100% of 500mhz that the machine is currently using.