Apache handles a thundering herd a little differently than you might imagine. When you get a burst of inbound traffic, it spawns a number of child processes, if it determines it needs more, it spawns twice as many in the next interval until it finally has enough processes to serve the requests or hits maxclients.
If you see these, it means that apache is just checking the children, and whatever caused apache to fork that many processes is probably gone. Yes they do take up client connections, but, whatever event caused things to spool up is probably gone.
First thing I would check in your logs would be a bunch of 302s prior to the event.
If you had something like
<?php include("http://www.oursite.com/header.php");?>
where header.php was missing and
ErrorDocument 404 /404.php
where 404.php included header.php, you would get a recursive loop and a hit on that page would immediately cause apache to use all available connections.
I'll start by admitting that I don't much about running stuff in clouds - but based on my experience elsewhere, I'd say that this webserver config reflects a fairly low volume of traffic. That the runqueue is so large suggests that there just isn't enough CPU available to deal with it. What else is in the runqueue?
We may be allowing far too many KeepAlive requests
No - keeplive still improves performance, modern browsers are very smart about knowing when to pipeline and when to run requests in parallel, although a timeout of 5 seconds is still rather high, and you've got a LOT of servers waiting - unless you've got HUGE latency problems I'd recommend cranking this down to 2-3. This should shorten the runqueue a little.
If you've not already got mod_deflate installed on the webserver - then I'd recommend you do so - and add the ob_gzhandler() to your PHP scripts. You can do this as an auto-prepend:
if(!ob_start("ob_gzhandler")) ob_start();
(yes, copression uses more CPU - but you should save CPU overall by getting servers out of the runqueue faster / handling fewer TCP packets - and as a bonus, your site is also faster).
I'd recommend setting an upper limit on MaxRequestsPerChild - say something like 500. This just allows some turnover on processes in case you've got a memory leak somewhere.
Your httpd processes look to be HUGE - make sure you've removed any apache modules you don't need and make sure you're serving up static content with good caching information.
If you're still seeing problems, then the problem is probably within the PHP code (if you switch to using fastCGI, this should be evident without any major performance penalty).
update
If the static content doesn't vary much across pages, then it might also be worth experimenting with:
if (count($_COOKIE)) {
header('Connection: close');
}
on the PHP scripts too.
Best Answer
Apart from recompiling mod_status to suite your need (that could sound a bit an overkill but.... it's still feasible), mod_status provide an option specifically designed for machine-readable processing. According to official documentation:
So to capture the output of mod_status is as simple as including a call to wget, curl or any other http client library than can be launched/included in your application, to suite your need.
Unfortunately I just discovered that when using the "?auto" format, most of the additional information provided by the ExtendedStatus directive are not displayed! This means that with the "?auto" option, you haven't access to the processlist.
As it sounded a little strange, I checked the source code of the mod_status module. Apart for an additional and not-documented "?notable" option, source code in "apache2-2.2.22/modules/generators/mod_status.c" (of my Ubuntu 12.04 LTS notebook) includes:
(BTW: I found both interesting and curious to read "?notable - Returns page for browsers without table support" as I'm so old/ancient to remember the early days of the web, where table support was a new feature of available browsers!)
I've also checked that missing processlist in the "?auto" format is a by-design feature:
As you can see, what you need is in the "else" part of the last "if". Hence, it's not included in the "?auto" format, as in this case we fall in the "short_report" case.
So, after all of the above and getting back to your question: "Is there any tweak, option or flag to hide those OPTIONS processes?", my answer is that your only option is to "tweak" a little application that:
As I'm comfortable with PERL and had some luck with the HTML::TableExtract module, a good basis that you could use is the following:
In my case, above script produce following output:
and you can see, skip the OPTIONS rows.
Please note that above application lacks basic error-handling so.... don't blame me if something will go wrong :-)