Detect Memory Leak on Linux – How to Identify Memory Usage Issues

linuxmemory leakmemory usageresourcestop

I have server which runs liquidsoap+icecast bundle and simple website (httpd+mysqld). Nothing special. Visitors around 2000+ per day, with around 50 being online simultaneously on average.

Server has 8GB RAM. As the time goes by, amount of free memory constantly decreases, although nothing new is started on server and there are no new users. At some point it starts to swap, load on server goes up and it becomes unresponsive. Usually what I do is just restart the server…

What can be done to detect what exactly leaks memory? I use top to monitor usage of resources, but as far as I see it shows nothing helpful:

enter image description here

Is there any way to find out what uses that much memory? or what starts to swap to disk heavily? Any way to free up memory without rebooting the server?

Best Answer

Running top in batch mode to report memory sizes periodically can be used to see who is using the memory when things go south. Runing sar in batch mode should give some good diagnostics on memory use, and related I/O. Running munin to monitor the system should give you a graph with good detail on what memory is being used for. This may help a lot.

You can use limits.conf to limit the maximum core size of programs. Properly set, this should kill any programs which are leaking memory. This works with the pam_limits module. Limits can also be set with the ulimits command.

You are running a few programs which could use large amounts of memory. Some things you could look at include.

  • Poorly programmed applications running under apache2 can leak memory. You should see the memory size increase when this happens. You can tune apache2 to recycle children after a certain number of uses by setting MaxRequestsPerChild to 100 or so. If this resolves the problem, then you need to resolve the leak. I would watch this first.
  • MySQL may try to load data into memory. If you have a lot of data in memory this may cause some thrashing, but should not be as dramatic as you are seeing.
  • If you have a large tmpfs file system mounted, then you may leak memory if files are not deleted when used. Large long lived files can also be a problem.
  • If the problem is occurs at roughly the same time of day, you may have a scheduled program which is leaking memory.
  • If you have a program that allocates shared memory, but does not release it before exiting, you will have a relatively invisible memory leak. If the shared memory is locked in memory, then it may force swapping. The amount of available shared memory is typically relatively limited.
  • The liquidsoap+icecast bundle could run into buffering issues that use memory. I haven't used this combination, so I am not sure how this would appear.

Normal memory usage: Free memory is not something you want a lot of. If your system has been up for a long time and has a lot of free memory something is wrong. Every time you read or write a file, the blocks will go into the buffer cache. This will reduce your free memory, and is a good thing. The system will keep enough free space to start a few programs without looking elsewhere for memory. As many programs run quickly, their memory will be returned to the free pool when they stop running.

When you read a file that is in buffer cache, no disk access is required and the read is resolved from the buffer cache. Writes use a similar mechanism. If your system needs memory, the buffer cache is one of the first places that is used. Most buffers can be released immediately.

If you have a memory leak, you will see free memory and buffers both begin to shrink. This is still not a severe problem, as the leaked memory should eventually be moved to swap space. Your system will still run fine until you fill the swap space, and draw down the remaining free space to the point programs can't be started. It is typical that a small amount of swap space may be used.

Related Topic