How to test VM monitoring in vmware ESXi

A couple of things:

1. Don't use Limits on RAM. To cap the amount of RAM the VM can use, use the amount of RAM when definining the VM. Otherwise, if you set a limit for RAM on a VM smaller than the amount of RAM that's defined for the VM, ESX will be forced to page. If paging is to occur, you want the guest to page inside itself because it's more intelligent about what data should be paged than ESXi is. When definining a VM, you want to define the RAM as the highest amount of RAM the VM might need, or the cap you want the VM to use.

2. Unless absolutely necessary, do not use reservations for any VM on any resource, including memory. If you install VMTools, which you always should unless you have a good reason not to, ESXi can reclaim memory, particularly when it's not being used. Setting reservations will force ESXi to provide physical memory even if the guest doesn't need it. Instead, use Shares to set priority when there is resource contention.

The above is very important. If you do not follow those, what will happen is you'll have inefficient paging, and inefficient granting of resources because VM's that don't need RAM will get it anyway, and VM's that could use the RAM won't get it. You'll also often times have VM's that need RAM, but may not get it even if there is some to provide.

The difference between those two screens comes down to the summary being how much RAM is actually being used, and total capacity shows the amount that could be used adjusted after reservations.

To answer your question, this is a reservation setting for the Hypervisor. This illustrates actually my above point that you shouldn't use reservations unless you abslutely have to. The ESXi hypervisor has perhaps too high of a memory reservation. You can tweak this. Reference the following article:

http://vm-help.com/esx40i/memory_allocation.php

I don't recommend doing that for production though.

You can effectively prevent fork bombs even on a physical machine with user limits. On FreeBSD, one way to configure this is in /etc/login.conf, with the maxproc parameter.

For more infos, refer to the documentation here.

Edit: If this isn't sufficient, you should be able to keep at least the XenServer running with limiting the CPU time any VM can get. See the sections on VCPU params in the Xen CLI doc here.