Thanks everyone for your answers! Following your suggestions I've been able to reduce my memory usage to 195M SWAP and 108M RSS, without touching my code (I'll definitely optimize it soon, but this was supposed to be a solution to get me out of trouble fast).
Here's the list of things I did:
Got rid of the wildcard used in VirtualHost entries. Instead of *:80 and *:443, I used the real IP of my server.
Changed Apache's prefork MPM. These are the values I ended up using:
StartServers 1
MinSpareServers 1
MaxSpareServers 5
ServerLimit 16
MaxClients 16
MaxRequestsPerChild 0
ListenBacklog 100
These are by no means magical numbers. I've spent some time trying different values and combination, and then testing them against the real usage of my server and everyone should do the same in their enviroment. For the record, my server receives close to 2M pvs/month, serving both dynamic pages and assets at a regular rate - no digg effect. The intention, again, was to reduce the memory footprint, not to improve performance or HA.
Reference:
Tunned down Apache's KeepAlive. By setting KeepAliveTimeout to a lower value (2 in my case) I can expect less server processes just waiting on connections with idle clients that may not request any more content.
Reference: http://httpd.apache.org/docs/2.0/mod/core.html#keepalivetimeout
Removed MySQL's unused module. I added skip-innodb to MySQL's my.cnf. Massive memory consumption reduction.
There are also some remarkable good suggestions that I couldn't personally do:
- Remove PHP modules you do not need. The PHP on my server has most mods already compiled, I'll probably try my own minimal PHP on other VPS.
- Switch to nginx with php-fastcgi. That's another good advice that I'll be trying soon, but right now I can't risk the downtime.
You have to pay for vSphere with its various modules and extra features but not to use the vSphere Client to connect to a free ESXi.
I think where you may be getting the license message from is although ESXi is free, you still need to request a free license key from VMWare.
Login to your ESXi box with vSphere Client and go to Configuration -> Licensed Features -> Edit.
If you are set to evaluation mode, that is what you are getting the license warning from.
VMWare should have emailed you a license key when you signed up on their website to download ESXi. If not, you can go through the download steps again and the license key should be on one of the pages.
For me, if I go to https://www.vmware.com/products/esxi/ hit Download, login with my free VMWare account, then on the page with all of the download links, at the top of the list is my ESXi License.
The reason you are seeing the license message about vSphere is that in the Evaluation mode, some of the extra features that are only available with vSphere are enabled, once you enter a free ESXi license, those will be disabled and you won't get prompted anymore.
Also, you can use the vCenter Converter in the standalone mode (runs off of your workstation) for free with ESXi. This tool is immensely useful for moving VMs on and off of ESXi. http://www.vmware.com/products/converter/.
Best Answer
Ballooning only happens under some very specific conditions. For VMware ESXi to "inflate" the balloon driver in a VM, the host must be under very heavy memory contention, commonly less than 4% of free memory on the host.
The idea with the balloon driver is for VMTools to start consuming memory inside the VM, to which the OS of the VM responds to by giving the balloon driver memory pages. The balloon driver doesn't actually need these pages for anything, but it tells the hypervisor which memory pages the VM doesn't actually use for anything, so they can be safely discarded by the hypervisor. If this doesn't solve the memory contention problem on the host, the hypervisor continues to inflate the balloon driver, up to 65% percent of the assigned virtual memory by default, which causes the OS on the VM to start swapping to disk. This is not a great outcome, but it's a lot better than the hypervisor swapping to disk instead as the OS on the VM actually knows what pages makes sense to swap to disk.
The way to limit ballooning is to either turn it off per VM (not a great strategy), or to reserve memory for a VM. If a VM has say 4 GB of RAM and a 3 GB memory reservation, the balloon driver can only inflate to 1 GB.
As ballooning is a bit of a last ditch strategy for ESXi, it almost always means that you have too little memory installed on your host.