(1) I see that each of the running processes occupies a very small percentage of memory (%MEM no more than 0.2%, and most just 0.0%), but how the total memory is almost used as in the fourth line of output ("Mem: 130766620k total, 130161072k used, 605548k free, 919300k buffers")? The sum of used percentage of memory over all processes seems unlikely to achieve almost 100%, doesn't it?
To see how much memory you are currently using, run free -m. It will provide output like:
total used free shared buffers cached
Mem: 2012 1923 88 0 91 515
-/+ buffers/cache: 1316 695
Swap: 3153 256 2896
The top row 'used' (1923) value will almost always nearly match the top row mem value (2012). Since Linux likes to use any spare memory to cache disk blocks (515).
The key used figure to look at is the buffers/cache row used value (1316). This is how much space your applications are currently using. For best performance, this number should be less than your total (2012) memory. To prevent out of memory errors, it needs to be less than the total memory (2012) and swap space (3153).
If you wish to quickly see how much memory is free look at the buffers/cache row free value (695). This is the total memory (2012)- the actual used (1316). (2012 - 1316 = 696, not 695, this will just be a rounding issue)
(2) how to understand the load average on the first line ("load average: 14.04, 14.02, 14.00")?
This article on load average uses a nice traffic analogy and is the best one I've found so far: Understanding Linux CPU Load - when should you be worried?. In your case, as people pointed out:
On multi-processor system, the load is relative to the number of processor cores available. The "100% utilization" mark is 1.00 on a single-core system, 2.00, on a dual-core, 4.00 on a quad-core, etc.
So, with a load average of 14.00 and 24 cores, your server is far from being overloaded.
The quick solution is to use MaxRequestsPerChild (number) (for example, 10000) to have the Apache restart each worker after that many requests. That will discard the memory used when it's restarted.
The 276m number isn't how much each process is using though. An explanation of the values shown in 'top' is helpful here:
VIRT: Virtual Image (kb)
The total amount of virtual memory used by the task. It includes all code, data and shared libraries plus pages that have been swapped out. (if you are using APC, the memory space used by it will also be included in this value)
RES: Resident size (kb)
The non-swapped physical memory a task has used.
SHR: Shared Mem size (kb)
The amount of shared memory used by a task. It simply reflects memory that could be potentially shared with other processes.
In 'top', you can add a 'data' column
Data: Data+Stack size (kb)
The amount of physical memory devoted to other than executable code, also known as the ’data resident set’ size or DRS.
That 'Data' value more closely matches the unique memory being used by that particular process, which is probably not that much. Adding up those 276M and getting a number near to 2GB means that you're double-counting a lot of things.
Best Answer
The virtual memory size isn't as important as your resident set. Virtual memory is going to include things that are mapped in such as shared libraries and whatnot (which will only be resident in physical RAM once). RSS tells you what's actually in memory, where as Virt. tells you how much of the available virtual address space has been swallowed.
Your second calculation is going to be closer, though it's quite low. A server with 4GB of RAM can run far more than 256 Apache processes. Depending on your traffic patterns and IO wait limitations, running more than 256 may be a good idea as a good number of those processes may just be sitting around waiting for the kernel to shovel data from device to device. Also factor in things like COW & the fact that everything points to the same 'httpd' binary, and you get more efficiencies.
Now, go and rebuild that system of yours with a good 2GB of swap space. Swap doesn't just act as "slow memory" anymore.
As a disclaimer, It's been a long time since I've cared about the specifics of Linux memory management, and might be off a bit on my facts, but the gist is solid!