(1) I see that each of the running processes occupies a very small percentage of memory (%MEM no more than 0.2%, and most just 0.0%), but how the total memory is almost used as in the fourth line of output ("Mem: 130766620k total, 130161072k used, 605548k free, 919300k buffers")? The sum of used percentage of memory over all processes seems unlikely to achieve almost 100%, doesn't it?
To see how much memory you are currently using, run free -m. It will provide output like:
total used free shared buffers cached
Mem: 2012 1923 88 0 91 515
-/+ buffers/cache: 1316 695
Swap: 3153 256 2896
The top row 'used' (1923) value will almost always nearly match the top row mem value (2012). Since Linux likes to use any spare memory to cache disk blocks (515).
The key used figure to look at is the buffers/cache row used value (1316). This is how much space your applications are currently using. For best performance, this number should be less than your total (2012) memory. To prevent out of memory errors, it needs to be less than the total memory (2012) and swap space (3153).
If you wish to quickly see how much memory is free look at the buffers/cache row free value (695). This is the total memory (2012)- the actual used (1316). (2012 - 1316 = 696, not 695, this will just be a rounding issue)
(2) how to understand the load average on the first line ("load average: 14.04, 14.02, 14.00")?
This article on load average uses a nice traffic analogy and is the best one I've found so far: Understanding Linux CPU Load - when should you be worried?. In your case, as people pointed out:
On multi-processor system, the load is relative to the number of processor cores available. The "100% utilization" mark is 1.00 on a single-core system, 2.00, on a dual-core, 4.00 on a quad-core, etc.
So, with a load average of 14.00 and 24 cores, your server is far from being overloaded.
Debian bug 666021 seems to be a report of this same issue. The suggestion there is:
#change value for this boot
sysctl -w vm.min_free_kbytes=65536
#change value for subsequent boots
echo "vm.min_free_kbytes=65536" >> /etc/sysctl.conf
http://russ.garrett.co.uk/2009/01/01/linux-kernel-tuning/ has some discussion of when altering this setting may be useful, reproduced here:
This tells the kernel to try and keep 64MB of RAM free at all times.
It’s useful in two main cases:
Swap-less machines, where you don’t want incoming network traffic to overwhelm the kernel and force an OOM before it has time to flush
any buffers.
x86 machines, for the same reason: the x86 architecture only allows DMA transfers below approximately 900MB of RAM. So you can end
up with the bizarre situation of an OOM error with tons of RAM free.
I applied this setting on my 3.2.12-gentoo x86 machine, but I'm still getting these errors.
Best Answer
pdflush's primary duty is flushing the disk buffer cache (AFAIk this is its only duty, but a Linux guru may correct me on this). The cache it flushes includes files that have been written to but not yet committed to disk.If I recall correctly
kswapdis the virtual process responsible for dealing with swap space & shuffling memory pages from RAM to disk & back again.For the health of your system, please set up a swap partition for your server (or
swaponthe one you have)Unix-like systems expect to be able to swap. Bad Things can happen when there is no available swap (like the disk buffer cache getting squeezed down to a tiny bit of memory, which can really make
pdflushhave a bad day).Even if you'll "never" use the swap space, disk is cheap enough that you can throw 2G at it.