Extremely high apache 2 initial response time, no system load

apache-2.2latency

I've noticed recently that a VPS that I administrate has some strange Apache behaviour. Essentially, any HTTP request hangs for about a second before it is actually served.

Output from ab shows something like this:

$ ab -n 10 -c 1 http://server-public-ip/path-to-static-file
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
<snipped>

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   3.5      0      11
Processing:     2 1373 1784.6   1000    4854
Waiting:        2 1372 1784.6   1000    4854
Total:          2 1374 1784.4   1011    4854

<snipped>

My initial thought was that my mod_perl website was running slowly, but this is running on a statically-served file! My virtual host config looks like:

<VirtualHost server-ip:80>
    <Location />
            DefaultType text/html
            Order deny,allow
            Allow from all
            SetHandler perl-script
            PerlHandler HTML::Mason::ApacheHandler
            PerlSetVar MasonDataDir "/var/tmp/mason-cache"
            PerlSetVar MasonCompRoot "<path-to-comproot>"
            ReWriteEngine on
            ReWriteRule ^(.*)/$ $1/index
            ErrorDocument 404 /not_found
    </Location>

    <Location /static>
            Order deny,allow
            Allow from all
            SetHandler default-handler
    </LocationMatch>
</VirtualHost>

There is plenty of memory free (at least 32MB entirely free, i.e. not used for cache), lots of processor resources (load average is 0.00 on 1/5/15-minute windows), lots of disk I/O available (median disk transfer is less than 5 KB/s) . . . I've no idea what the problem might be. After the transfer begins it runs at full speed (more than 2MB/s, which is about the limit of my local connection.)

I'm using Debian stable, running a 2.6.32 kernel and Apache 2.2.16 (using prefork). There is little to no traffic going to the server, and I've tried tweaking the prefork settings in both directions (lots of spare workers to no spare workers).

Any thoughts?

Best Answer

Keep in mind that even if you have separate handlers for different paths, the prefork children get reused based on your MaxRequestsPerChild setting. This means that a child may end up servicing a request to the perl handler before it services one to the static content (especially if you've got a load balancer with a health check against the perl handler). If there's a quirk in the Perl that is gumming up the works, it can gum up the static content on subsequent requests.

For this and other reasons, at a prior workplace we had a practice of isolating the static content to its own httpd process and using the load balancer to split traffic between the two based on URI context. The second webserver never even loads mod_perl, isolating any problems introduced by code to the other process.

Related Topic