Python – Apache mod_wsgi tuning

apache-2.4djangomod-wsgipython

I have a django site on webfaction that uses apache + mod_wsgi.

Site is getting around 1000 requests per minute.

But it makes some calculations, so request takes about 5-10 seconds.

I use the following configuration

StartServers         2
MinSpareThreads      10
MaxSpareThreads      25
ThreadLimit          25
ThreadsPerChild      25
MaxClients           75
MaxRequestsPerChild   1000

threads=15 processes=12

The problem is high CPU usage and it takes time to process a simple static page without calculations (looks like Apache queued the request).

So what I want is for Apache to quickly accept requests.

I'm totally lost because of number of parameters, I also don't quite understand what they mean. What do we need StartServers and MaxRequestWorkers for?

Any help and/or explanations will be highly appreciated.

I have 8GB of RAM.

Apache MPM Worker.

mod_wsgi 4.4.21.

Thank you in advance.

Best Answer

StartServers is the number of number of server processes you start with, and MaxRequestWorkers is the number of threads per process. The Webfaction settings should be reasonable in most situations although a thousand requests a minute may need some tuning, but probably mostly in the application.

In normal use httpd would take a request, pass it to mod_wsgi and wait for its return, which should be near instantaneous, so what is actually taking its time is whatever the python script is doing. Your httpd worker threads are therefore in a wait state and will build up as requests come in while other requests are being processed, so even a static page will end up waiting if your threads are occupied.

Look at what the application is doing and for solutions. You may be able to cache queries using memcached or similar. If the time that your application takes to process a request is unavoidable, look at making it asynchronous using a message queue like Celery, so rather than having your web server wait for responses, you can poll for them using browser side scripting.

Splitting the static page serving from the dynamic will also improve the response. If possible you could run multiple sets of workers, or pass static page and object serving to nginx, which is a more common way of handling wsgi.

Another method would be to serve python through a native web server such as tornado or gunicorn and use apache as a reverse proxy, which may improve the backend response although still won't help if processes are causing large numbers of waiting threads.

Related Topic