(1) I see that each of the running processes occupies a very small percentage of memory (%MEM no more than 0.2%, and most just 0.0%), but how the total memory is almost used as in the fourth line of output ("Mem: 130766620k total, 130161072k used, 605548k free, 919300k buffers")? The sum of used percentage of memory over all processes seems unlikely to achieve almost 100%, doesn't it?
To see how much memory you are currently using, run free -m. It will provide output like:
total used free shared buffers cached
Mem: 2012 1923 88 0 91 515
-/+ buffers/cache: 1316 695
Swap: 3153 256 2896
The top row 'used' (1923) value will almost always nearly match the top row mem value (2012). Since Linux likes to use any spare memory to cache disk blocks (515).
The key used figure to look at is the buffers/cache row used value (1316). This is how much space your applications are currently using. For best performance, this number should be less than your total (2012) memory. To prevent out of memory errors, it needs to be less than the total memory (2012) and swap space (3153).
If you wish to quickly see how much memory is free look at the buffers/cache row free value (695). This is the total memory (2012)- the actual used (1316). (2012 - 1316 = 696, not 695, this will just be a rounding issue)
(2) how to understand the load average on the first line ("load average: 14.04, 14.02, 14.00")?
This article on load average uses a nice traffic analogy and is the best one I've found so far: Understanding Linux CPU Load - when should you be worried?. In your case, as people pointed out:
On multi-processor system, the load is relative to the number of processor cores available. The "100% utilization" mark is 1.00 on a single-core system, 2.00, on a dual-core, 4.00 on a quad-core, etc.
So, with a load average of 14.00 and 24 cores, your server is far from being overloaded.
The cache memory in Linux-based machines is often used to cache disk blocks into memory to prevent writing to the harddisk and speed up I/O operations. Also, a lot of free memory is cached and released when it is actually required.
When interpreting the output of free, you should look at the second row for the actual memory usage.
The above numbers can be interpreted as:
- 9534176 kilobytes cached memory
- 737752 actually used memory
- 15701956 actually free memory
For more information, take a look at http://www.faqs.org/docs/linux_admin/buffer-cache.html.
Best Answer
You're on a Virtuozzo or OpenVZ PVS (Pseudo-Virtual Server), I see, which makes the concept of "memory" a fairly specious one, at best. Your provider has limited the amount of "overcommitted" memory you can use, which you've bumped up against. That "total memory" number you see in free, by the way, has no relation to reality in a VZ PVS -- the provider can configure VZ to show whatever number they want in there, regardless of what resources have actually been allocated to you.
You need to lodge this support request with your provider, as they're the only ones who can help you (we can't see what the configuration for your PVS is, let alone change it), and if you don't get the support you need from them, I'd highly recommend switching to a provider that can give you (a) a proper VPS that actually has the memory available that it says it does (ie. NOT a VZ-based PVS), and (b) proper support for what you've paid for.