Linux – How to have linux top command show memory summary in megabytes not kilobytes

Running top in batch mode to report memory sizes periodically can be used to see who is using the memory when things go south. Runing sar in batch mode should give some good diagnostics on memory use, and related I/O. Running munin to monitor the system should give you a graph with good detail on what memory is being used for. This may help a lot.

You can use limits.conf to limit the maximum core size of programs. Properly set, this should kill any programs which are leaking memory. This works with the pam_limits module. Limits can also be set with the ulimits command.

You are running a few programs which could use large amounts of memory. Some things you could look at include.

• Poorly programmed applications running under apache2 can leak memory. You should see the memory size increase when this happens. You can tune apache2 to recycle children after a certain number of uses by setting MaxRequestsPerChild to 100 or so. If this resolves the problem, then you need to resolve the leak. I would watch this first.
• MySQL may try to load data into memory. If you have a lot of data in memory this may cause some thrashing, but should not be as dramatic as you are seeing.
• If you have a large tmpfs file system mounted, then you may leak memory if files are not deleted when used. Large long lived files can also be a problem.
• If the problem is occurs at roughly the same time of day, you may have a scheduled program which is leaking memory.
• If you have a program that allocates shared memory, but does not release it before exiting, you will have a relatively invisible memory leak. If the shared memory is locked in memory, then it may force swapping. The amount of available shared memory is typically relatively limited.
• The liquidsoap+icecast bundle could run into buffering issues that use memory. I haven't used this combination, so I am not sure how this would appear.

Normal memory usage: Free memory is not something you want a lot of. If your system has been up for a long time and has a lot of free memory something is wrong. Every time you read or write a file, the blocks will go into the buffer cache. This will reduce your free memory, and is a good thing. The system will keep enough free space to start a few programs without looking elsewhere for memory. As many programs run quickly, their memory will be returned to the free pool when they stop running.

When you read a file that is in buffer cache, no disk access is required and the read is resolved from the buffer cache. Writes use a similar mechanism. If your system needs memory, the buffer cache is one of the first places that is used. Most buffers can be released immediately.

If you have a memory leak, you will see free memory and buffers both begin to shrink. This is still not a severe problem, as the leaked memory should eventually be moved to swap space. Your system will still run fine until you fill the swap space, and draw down the remaining free space to the point programs can't be started. It is typical that a small amount of swap space may be used.

It's a case of top reporting memory usage per process as if it where the only process running. In reality all the apache processes have shared memory (common linked libraries), and this shared memory is being reported for each process so the total usage across all apache processes appears to be more than it really is.

There are numerous posts that discuss how to underdstand and interpret the output of top and ps.

process memory usage on Linux is a complex matter; you can't just run ps and know what is going on. This is especially true when you deal with programs that create a lot of identical children processes, like Apache. ps might report that each Apache process uses 10 megabytes of memory, when the reality might be that the marginal cost of each Apache process is 1 megabyte of memory