One of the apache processes is huge – how can I find out why

Tracking down WHAT is causing the problem can be a pain in the ass. The first thing I'd do if I had a problem like that is reduce MaxRequestsPerChild to an aggresively low number (~100-200) and see if that makes a difference. If it does, then you probably have code that is leaking memory in a loop somewhere and you'll want to run a code audit.

Another thing to look at is Apache's fullstatus, see if you can find out what particular request is causing the memory leak. Get the PIDs on your suspected processes and run an strace on them.

Enable mod_headers in Apache and then add to your VirtualHost:

RequestHeader add X-Queue-Start "%t"

New Relic will then show you queueing time in the main overview chart.

This is the time between when request is first accepted by Apache child worker process and when mod_wsgi daemon process gets to handle the request. This can be used as one indicator of requests getting backlogged which in turn can indicate thread starvation in daemon process due to deadlocked threads or threads waiting on an external resource.

Unfortunately New Relic relies on requests completing to report data for that request. So if a request gets stuck you will not know about it. If all threads get stuck then daemon process will stop handling more requests.

Problem is that if number of processes/threads across Apache child worker processes is less than 100, the daemon process listener backlog, then all those threads can also get stuck and you will not know from Apache error logs because they will just sit there waiting for daemon to accept connection which never happens. Only the HTTP browser client will know as it will get connection refused when Apache child worker socket backlog fills up.

In mod_wsgi 4.0 I will be adding ability to configure the listener backlog for daemon process so can be reduced so you might get an error of some sort. There are already new options in mod_wsgi 4.0 to look for blocked threads and to restart daemon processes automatically and also dump a stack trace of where blocked threads were in code at the time.

To get that you would need to use mod_wsgi 4.0 dev code from mod_wsgi repo. You could then set blocked-timeout on WSGIDaemonProcess to 60 seconds and when all threads get stuck it will do the restart and recover, plus dump stack traces. Am still tweaking this and there are other configuration options related to this why don't describe here.

The mod_wsgi 4.0 code also has some other features which can be used with custom charting in New Relic to track growing number of blocked threads. Am not happy with it and needs to be changed a bit, but is stable.

Anyway, jump onto the mod_wsgi mailing list to discuss further.