Mod_fcgi in virtualmin: graceful kill fail, sending SIGKILL

apache-2.2mod-fcgid

Yesterday around 1am, our server ground to a crawl. This doesn't happen often, but I'm trying to get to the bottom of it.

There is no unusual traffic volume, no unusual processes running, just all of the sudden the server started killing fcgid processes.

[Thu Aug 02 01:17:32 2012] [warn] mod_fcgid: process 26460 graceful kill fail, sending SIGKILL

… for as many fcgid processes as we have…

CPU idle fell to 0% and I/O seemed to take up most of the load. The issue lasted about 5 minutes.

I suspect there was some swap activity, although I'm not sure if it was due to killed processes being swapped in to die, or if it was because some process ramped up memory usage faster than my process watching scripts can see them.

The oom-killer wasn't triggered (at least it's not logged), so I think this was Apache for some reason restarting the processes. This is not regular, and nothing obvious appears in cron.

Is there a normal Apache process which might cause this? We run dozens of different sites, and it was late at night, so volume was very, very low. (maybe 200 requests in a 10 minute period).

Best Answer

Modify this file /etc/httpd/conf.d/fcgid.conf and change, FcgidIOTimeout to

FcgidIOTimeout 90

It works for me.

Tkx JD

Related Solutions

Linux – How to stop Apache from crashing the entire server

I hate to say it, but you appear to be asking the wrong question.

It's not about stopping Apache from bringing down your server, it's about having your webserver serve more queries per second - enough so that you don't have a problem. A part of the answer to the reframed question is then limiting Apache so that it does not crash at high loads.

For the second part of that, Apache has some limits you can set - MaxClients being an important configuration. This limits how many children it's allowed to run. If you can take load off Apache for long-running processes (large files being downloaded for example), that's another slot in Apache to be able to serve PHP. If the file downloads have to be verified by the PHP layer, they can still do that, and pass back out to a more optimised webserver for the static content, such as with NginX sendfile

Meanwhile, forking Apache every on every single request for the slowest way to run PHP - as a CGI (whatever apache MPM you may be using) - is also having the machine spend large amounts of time not running your code. mod_php is significantly more optimised.

PHP can do huge amounts of traffic when Apache and the PHP layer are appropriately optimised. Yesterday, 11th Dec 2010, for example, the pair of PHP servers that I run did almost 19 Million hits in the 24hr period, and most of that in the 7am-8pm time-period.

There are plenty of other questions here, and articles elsewhere about optimising Apache and PHP, I think you need to read them first, before blaming Linux/Apache & PHP.

Httpd – Apache: graceful kill fail, sending SIGKILL. How much is too much

Other people have reported a similar issue with FCGID processes hanging and being unkillable in other systems, such as Wordpress and Sympa.

A suggested fix was to add the option

FcgidMaxRequestLen 1000000000

to the Apache configuration (default for this setting is 131072). I have not been able to verify this fixes the issue but there is anecdotal evidence to indicate that it does.

Related Topic