General rule of thumb -
Run this command when server is under load:
ps -ylC httpd --sort:rss | awk '{sum+=$8; ++n} END {print "Tot="sum"("n")";print "Avg="sum"/"n"="sum/n/1024"MB"}'
That will tell you approximately the average size of an Apache process.
This is not a dedicated web node so say 60% of the RAM is available for Apache.
4096 * .60 / AVERAGE_SIZE_HTTPD_PROCESS = approximately the # of MaxClients/requests you can service. Generally looks like your average is around 170MB so...
You can service 14 requests at a time. Reduce MaxClients to a more reasonable number like 20-25.
Cheers
We know the memory problem is coming from apache/PHP because whenever we issue a /etc/init.d/httpd reload the memory usage drops
No - that just means it's related to the web traffic. You've gone on to mention that you're running mysql on the box - presumably managing data for the webserver - it could just as easily be the culprit here. As could other services your webstack uses which you've not mentioned.
Each apache thread is assigned a PHP memory_limit of 512MB which explains
No it doesn't. You're reporting an average of 7 and a max of 25 busy servers - yet your memory graph shows a delta of around 25Gb.
Really you should start again with basic HTTP tuning - you seem to be running a constant 256 httpds, yet your peak usage is 25 - this is just plain dumb.
and a max_execution_time of 120 sec which should terminate threads which execution is taking longer
No - only if the thread of execution is within the PHP interpreter - not if PHP is blocked.
that performs financial modeling
(sigh)
It would have been helpful if you'd provided details of how you have configured Apache, threaded or prefork, what version, how PHP is invoked (module, cgi, fastcgi), whether you are using persistent connections, whether you use stored procedures.
I'd suggest you start by moving mysql onto a seperate machine and stop using persistent connections (if you're currently using them). Set the memory limit much lower and override this on a per-script basis. Make sure you've got the circular reference garbage collector installed and configured.
Best Answer
you can clear a cache of linux machine and pu it into a crontab entry to run in regular interval.