Linux – Apache strace to hunt down a memory leak

apache-2.2centosdrupallinuxstrace

We have a server with a memory issue: the server keeps allocating itself memory and doesn't release it. We're running Apache. I set MaxReqsPerClient to a really low value just so the threads don't hold a lot of memory, but has anyone seen calls like this? Am I wrong in thinking that it's probably Drupal pulling too much data back from the cache in DB?

read(52, "h_index\";a:2:{s:6:\"weight\";i:1;s"..., 6171) = 1368
read(52, "\";a:2:{s:6:\"author\";a:3:{s:5:\"la"..., 4803) = 1368
read(52, ":\"description\";s:19:\"Term name t"..., 3435) = 1368
read(52, "abel\";s:4:\"Name\";s:11:\"descripti"..., 2067) = 1368
read(52, "ions\";a:2:{s:4:\"form\";a:3:{s:4:\""..., 16384) = 708
brk(0x2ab554396000)                     = 0x2ab5542f5000
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ab55f653000
brk(0x2ab554356000)                     = 0x2ab5542f5000
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ab55f753000
brk(0x2ab554356000)                     = 0x2ab5542f5000
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ab55f853000
brk(0x2ab554356000)                     = 0x2ab5542f5000
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ab55f953000
brk(0x2ab554356000)                     = 0x2ab5542f5000
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ab55fa53000
brk(0x2ab554356000)                     = 0x2ab5542f5000
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ab55fb53000
brk(0x2ab554356000)                     = 0x2ab5542f5000
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ab55fc53000
poll([{fd=52, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(52, "d\0\0\0\3SELECT cid, data, created, "..., 104) = 104
read(52, "\1\0\0\1\5E\0\0\2\3def\23drupal_database_nam"..., 16384) = 1368
read(52, ";s:11:\"granularity\";a:5:{s:4:\"ye"..., 34783) = 1368
read(52, ":4:\"date\";}s:9:\"datestamp\";a:9:{"..., 33415) = 1368
read(52, "\";i:0;s:15:\"display_default\";i:0"..., 32047) = 1368
read(52, "e as an integer value.\";s:8:\"set"..., 30679) = 1368
read(52, "label' pairs, i.e. 'Fraction': 0"..., 29311) = 1368

top (the procs just keep growing in memory..):

12845 apache    15   0  581m 246m  37m S  0.0  4.1   0:17.39 httpd                                   
12846 apache    15   0  571m 235m  37m S  0.0  4.0   0:12.13 httpd                                   
12833 apache    15   0  420m 117m  37m S  0.0  2.0   0:06.04 httpd                                   
12851 apache    15   0  412m 113m  37m S  0.0  1.9   0:05.32 httpd                                   
13871 apache    15   0  409m 109m  37m S  0.0  1.8   0:04.90 httpd                                   
12844 apache    15   0  407m 108m  37m S  0.0  1.8   0:04.50 httpd                                   
13870 apache    15   0  407m 108m  37m S  0.3  1.8   0:03.50 httpd                                   
14903 apache    15   0  402m 103m  37m S  0.3  1.7   0:01.29 httpd                                   
14850 apache    15   0  397m 100m  37m S  0.0  1.7   0:02.08 httpd                                   
14907 apache    15   0  390m  93m  36m S  0.0  1.6   0:01.32 httpd                                   
13872 apache    15   0  386m  91m  37m S  0.0  1.5   0:03.13 httpd                                   
12843 apache    15   0  373m  81m  37m S  0.0  1.4   0:02.51 httpd                                   
14901 apache    15   0  370m  75m  33m S  0.0  1.3   0:00.78 httpd                                   
14904 apache    15   0  335m  29m  15m S  0.0  0.5   0:00.26 httpd

Best Answer

Over what time period is this occurring? How big does memory usage get?

If it's a short-enough time, or if you don't running Apache in the foreground for a bit, you could use valgrind to find memory leaks... (It's available via YUM)

Something like:

valgrind --leak-check=full /usr/sbin/httpd

The output could potentially show something like:

==945== LEAK SUMMARY:
==945== definitely lost: 15,595 bytes in 120 blocks
==945== indirectly lost: 1,917,248 bytes in 1,242 blocks
==945== possibly lost: 0 bytes in 0 blocks
==945== still reachable: 11,743 bytes in 89 blocks
==945== suppressed: 0 bytes in 0 blocks
==945== Reachable blocks (those to which a pointer was found) are not shown.
==945== To see them, rerun with: --leak-check=full --show-reachable=yes

You can view the memory allocation of a process with the pmap utility.

For Apache, I'd take a look at the PID. Obtain the PID value with: cat /var/run/httpd.pid or just run:

pmap -x $(cat /var/run/httpd.pid)

But in my limited Apache experience, leaks always seem to be tied to modules and scripting languages. Could be caching, too...

Related Topic