The issue was that I had installed the package apache2-mpm-itk instead of apache2-mpm-prefork. apache2-mpm-itk is derived from apache2-mpm-prefork, but for some reason, didn't perform well when used with mod_wsgi.
We know the memory problem is coming from apache/PHP because whenever we issue a /etc/init.d/httpd reload the memory usage drops
No - that just means it's related to the web traffic. You've gone on to mention that you're running mysql on the box - presumably managing data for the webserver - it could just as easily be the culprit here. As could other services your webstack uses which you've not mentioned.
Each apache thread is assigned a PHP memory_limit of 512MB which explains
No it doesn't. You're reporting an average of 7 and a max of 25 busy servers - yet your memory graph shows a delta of around 25Gb.
Really you should start again with basic HTTP tuning - you seem to be running a constant 256 httpds, yet your peak usage is 25 - this is just plain dumb.
and a max_execution_time of 120 sec which should terminate threads which execution is taking longer
No - only if the thread of execution is within the PHP interpreter - not if PHP is blocked.
that performs financial modeling
(sigh)
It would have been helpful if you'd provided details of how you have configured Apache, threaded or prefork, what version, how PHP is invoked (module, cgi, fastcgi), whether you are using persistent connections, whether you use stored procedures.
I'd suggest you start by moving mysql onto a seperate machine and stop using persistent connections (if you're currently using them). Set the memory limit much lower and override this on a per-script basis. Make sure you've got the circular reference garbage collector installed and configured.
Best Answer
@raj you'll probably find that the processes are in an S state as they are waiting on another process (in a D state) bound by I/O ie. For the machine to stop swapping.
So the key is to address two key issues, standard memory allocation and memory commitment.
When running Magento, it is fairly easy to massively over commit on RAM simply by setting PHP Max memory limits too high. Under normal circumstances, it won't have any ill effects, but the moment the app tries to use that memory, it will almost immediately go into swap (if you've run out of RAM that is).
Then, you'll likely find that Apache isn't the main concern for memory usage, but rather MySQL is more likely the culprit.
You haven't mentioned your server specification or any specific details for your store, so it would be impossible to accurately advise. But simply revising your Apache and MySQL configuration would be an easy place to start. Drop the number of Apache threads, MySQL connections, allocated memory commitments. Then start graphing with a tool like Munin to actually be able to correlate traffic, threads, CPU usage and memory usage. Without doing so, you are working blind.
But I would advise to stop poking around netstat as you are a long way off needing to diagnose TCP states; you need to start from the beginning.