We're running a heavy Drupal website that performs financial modeling. We seem to be running into some sort of memory leak given the fact that overtime the memory used by apache grows while the number of apache processes remains stable:
We know the memory problem is coming from apache/PHP because whenever we issue a /etc/init.d/httpd reload
the memory usage drops (see above screenshot and below CLI outputs):
Before httpd reload
$ free total used free shared buffers cached Mem: 49447692 45926468 3521224 0 191100 22609728 -/+ buffers/cache: 23125640 26322052 Swap: 2097144 536552 1560592
After httpd reload
$ free total used free shared buffers cached Mem: 49447692 28905752 20541940 0 191360 22598428 -/+ buffers/cache: 6115964 43331728 Swap: 2097144 536552 1560592
Each apache thread is assigned a PHP memory_limit
of 512MB which explains the high memory usage depiste the low volume of requests, and a max_execution_time
of 120 sec which should terminate threads which execution is taking longer, and should therefore prevent the constant growth in memory usage we're seeing.
Q: How could we investigate what is causing this memory leak?
Ideally I'm looking for troubleshooting steps I can perform on the system without having to bother the dev team.
Additional info:
OS: RHEL 5.6
PHP: 5.3
Drupal: 6.x
MySQL: 5.6
FYI we're aware of the swapping issue which we're investigating separately and has nothing to do with the memory leak which we've observed before the swapping started to occur.
Best Answer
No - that just means it's related to the web traffic. You've gone on to mention that you're running mysql on the box - presumably managing data for the webserver - it could just as easily be the culprit here. As could other services your webstack uses which you've not mentioned.
No it doesn't. You're reporting an average of 7 and a max of 25 busy servers - yet your memory graph shows a delta of around 25Gb.
Really you should start again with basic HTTP tuning - you seem to be running a constant 256 httpds, yet your peak usage is 25 - this is just plain dumb.
No - only if the thread of execution is within the PHP interpreter - not if PHP is blocked.
(sigh)
It would have been helpful if you'd provided details of how you have configured Apache, threaded or prefork, what version, how PHP is invoked (module, cgi, fastcgi), whether you are using persistent connections, whether you use stored procedures.
I'd suggest you start by moving mysql onto a seperate machine and stop using persistent connections (if you're currently using them). Set the memory limit much lower and override this on a per-script basis. Make sure you've got the circular reference garbage collector installed and configured.