You have to pay for vSphere with its various modules and extra features but not to use the vSphere Client to connect to a free ESXi.
I think where you may be getting the license message from is although ESXi is free, you still need to request a free license key from VMWare.
Login to your ESXi box with vSphere Client and go to Configuration -> Licensed Features -> Edit.
If you are set to evaluation mode, that is what you are getting the license warning from.
VMWare should have emailed you a license key when you signed up on their website to download ESXi. If not, you can go through the download steps again and the license key should be on one of the pages.
For me, if I go to https://www.vmware.com/products/esxi/ hit Download, login with my free VMWare account, then on the page with all of the download links, at the top of the list is my ESXi License.
The reason you are seeing the license message about vSphere is that in the Evaluation mode, some of the extra features that are only available with vSphere are enabled, once you enter a free ESXi license, those will be disabled and you won't get prompted anymore.
Also, you can use the vCenter Converter in the standalone mode (runs off of your workstation) for free with ESXi. This tool is immensely useful for moving VMs on and off of ESXi. http://www.vmware.com/products/converter/.
A couple of things:
Don't use Limits on RAM. To cap the amount of RAM the VM can use, use the amount of RAM when definining the VM. Otherwise, if you set a limit for RAM on a VM smaller than the amount of RAM that's defined for the VM, ESX will be forced to page. If paging is to occur, you want the guest to page inside itself because it's more intelligent about what data should be paged than ESXi is. When definining a VM, you want to define the RAM as the highest amount of RAM the VM might need, or the cap you want the VM to use.
Unless absolutely necessary, do not use reservations for any VM on any resource, including memory. If you install VMTools, which you always should unless you have a good reason not to, ESXi can reclaim memory, particularly when it's not being used. Setting reservations will force ESXi to provide physical memory even if the guest doesn't need it. Instead, use Shares to set priority when there is resource contention.
The above is very important. If you do not follow those, what will happen is you'll have inefficient paging, and inefficient granting of resources because VM's that don't need RAM will get it anyway, and VM's that could use the RAM won't get it. You'll also often times have VM's that need RAM, but may not get it even if there is some to provide.
The difference between those two screens comes down to the summary being how much RAM is actually being used, and total capacity shows the amount that could be used adjusted after reservations.
To answer your question, this is a reservation setting for the Hypervisor. This illustrates actually my above point that you shouldn't use reservations unless you abslutely have to. The ESXi hypervisor has perhaps too high of a memory reservation. You can tweak this. Reference the following article:
http://vm-help.com/esx40i/memory_allocation.php
I don't recommend doing that for production though.
Best Answer
This appears to work: