Linux – Monitoring memory usage on Linux system

centoslinuxmemorymonitoringsnmp

We use WhatsUp Gold to monitor all of our web servers. On our Linux servers (and much to the same degree, our FreeBSD servers) I'm having a little bit of an issue with the memory monitors. We're using SNMP with WUG to grab the data from the servers. The memory counter that the SNMP daemon returns on the servers is the combined value (used, cached, buffers). Right now one of my servers looks like this:

[admin@stgwww snmp]$ free -m
             total       used       free     shared    buffers     cached 
Mem:          7872       1656       6216          0        143      1107
-/+ buffers/cache:        404       7467 
Swap:         4867          0       4867

The Value being returned via SNMP to WUG is 1656. From what I understand, the cached RAM is essentially FREE RAM with the added benefit of hanging on to data that previously occupied it in case it's needed again. So for our purposes of wanting to know how much RAM is actually being actively used, the value we're getting back is misleading. If we go off of what's being graphed by WUG, we're being led to believe that more RAM is being used and less is available than there really is.

So whats that best way to go about monitoring this? WUG allows me to write SSH scripts, which can SSH into the server every 5 minutes or so, execute a script and return the value (as long as it's a single numeric value). With this I've written a script that pulls the "404" number from the example above and divides it by the total amount giving me a percent used value which I return to WUG and graph on a chart that scales from 0 to 100. But this seems like way to much of a hack.

Am I better off monitoring the free+buffers+cached value? Is there a better way to do this in WUG? Thoughts?

Best Answer

Go and take a look at linuxatemyram.com. WUG is telling you what Linux thinks is used (used+buffers+cache). What you have decided to monitor (used/total) seems reasonable to me especially for a graph as it requires no knowledge of the system specifics.

Related Solutions

Linux – Caching/preloading files on Linux into RAM

vmtouch seems like a good tool for the job.

Highlights:

query how much of a directory is cached
query how much of a file is cached (also which pages, graphical representation)
load file into cache
remove file from cache
lock files in cache
run as daemon

EDIT: Usage as asked in the question is listed in example 5 on vmtouch Hompage

Example 5

Daemonise and lock all files in a directory into physical memory:

vmtouch -dl /var/www/htdocs/critical/

EDIT2: As noted in the comments, there is now a git repository available.

Linux – How to understand the memory usage and load average in linux server

(1) I see that each of the running processes occupies a very small percentage of memory (%MEM no more than 0.2%, and most just 0.0%), but how the total memory is almost used as in the fourth line of output ("Mem: 130766620k total, 130161072k used, 605548k free, 919300k buffers")? The sum of used percentage of memory over all processes seems unlikely to achieve almost 100%, doesn't it?

To see how much memory you are currently using, run free -m. It will provide output like:

             total       used       free     shared    buffers     cached
Mem:          2012       1923         88          0         91        515
-/+ buffers/cache:       1316        695
Swap:         3153        256       2896

The top row 'used' (1923) value will almost always nearly match the top row mem value (2012). Since Linux likes to use any spare memory to cache disk blocks (515).

The key used figure to look at is the buffers/cache row used value (1316). This is how much space your applications are currently using. For best performance, this number should be less than your total (2012) memory. To prevent out of memory errors, it needs to be less than the total memory (2012) and swap space (3153).

If you wish to quickly see how much memory is free look at the buffers/cache row free value (695). This is the total memory (2012)- the actual used (1316). (2012 - 1316 = 696, not 695, this will just be a rounding issue)

(2) how to understand the load average on the first line ("load average: 14.04, 14.02, 14.00")?

This article on load average uses a nice traffic analogy and is the best one I've found so far: Understanding Linux CPU Load - when should you be worried?. In your case, as people pointed out:

On multi-processor system, the load is relative to the number of processor cores available. The "100% utilization" mark is 1.00 on a single-core system, 2.00, on a dual-core, 4.00 on a quad-core, etc.

So, with a load average of 14.00 and 24 cores, your server is far from being overloaded.

Related Topic