Apache server status requests/sec

We had the same problem on Apache 2.4.6. After monitoring the server and adjusting the setting for several hours it appears to us that Apache may have a bug. What appears to happen is that the server processes occasionally goes into the G state (Gracefully finishing) and restarts to accept new requests, that's normal. What is not normal is that for some reason this can take up to a few minutes to restart. If you only have a few server process running and they all go into the G state at the same time then your scoreboard fills up and you won't be able to server any more requests.

What we did was increase the number of servers so there is a less of a chance that they will all go into the G state at the same time. Also make sure you allocate at least 25 threads (MaxRequestWorkers) for each server process because that appears to be the default (i.e. if 5 Servers x 25 ThreadsPerChild = 125 MaxRequestWorkers). You can change ThreadsPerChild if you like, we left it at default. If you don't allocate enough threads the additional servers will not start. We left MinSpareThreads at the default value which is 25 and the default for MaxSpareThreads which is 75. If you do modify these settings, the value for MaxSpareThreads must be greater than or equal to the sum of MinSpareThreads and ThreadsPerChild. Also MaxRequestWorkers must be equal to or less than the ServerLimit.

Here is what worked for us but it might not be the best configuration for you.

StartServers 3
MinSpareServers 5
MaxSpareServers 10
ServerLimit 250
MaxRequestWorkers 250
MaxConnectionsPerChild 1000
KeepAlive Off

Edit: This is a confirmed bug in httpd's mpm_event module which might not be fixable through configuration.
The linked bugtracker entry has a presumed patch and more discussion about how to fix this until a new version of the event module is officially released.

StartServers is the number of number of server processes you start with, and MaxRequestWorkers is the number of threads per process. The Webfaction settings should be reasonable in most situations although a thousand requests a minute may need some tuning, but probably mostly in the application.

In normal use httpd would take a request, pass it to mod_wsgi and wait for its return, which should be near instantaneous, so what is actually taking its time is whatever the python script is doing. Your httpd worker threads are therefore in a wait state and will build up as requests come in while other requests are being processed, so even a static page will end up waiting if your threads are occupied.

Look at what the application is doing and for solutions. You may be able to cache queries using memcached or similar. If the time that your application takes to process a request is unavoidable, look at making it asynchronous using a message queue like Celery, so rather than having your web server wait for responses, you can poll for them using browser side scripting.

Splitting the static page serving from the dynamic will also improve the response. If possible you could run multiple sets of workers, or pass static page and object serving to nginx, which is a more common way of handling wsgi.

Another method would be to serve python through a native web server such as tornado or gunicorn and use apache as a reverse proxy, which may improve the backend response although still won't help if processes are causing large numbers of waiting threads.