Apache Segmentation fault after logrotation on debian squeeze

apache-2.2debian-squeeze

Every sunday night during debian squeeze's logrotation, apache crashes when postlogrotation attemts to reload configuration, and/or restart apache. I've even tried reconfiguring the logrotation by hard restarting (stop start) the service, with the same faults.

What happends is that apache acctually seems to be starting fine, but every request throws the following error in apaches error.log:

[notice] child pid xxxx exit signal Segmentation fault (11)

If I restart apache manually once more, it works fine again. Does anyone have any idea why this is happening?

The AWS instance is running debian squeeze, apache 2.2.16, php 5.4.32 and varnish 3 (if it matters).

Any suggestions is very welcome. If theres relevant info I've missed, tell me!

UPDATE: I moved the Magento sites to another server, and the problem stopped occurring (even though there are still a few other sites left), and it didn't happen on the new server. Still few ideas as to what caused it.

Best Answer

There can be different reasons for such a failure and rather then assuming something, I would suggest that you take a little time to debug both the logrotate script and make a coredump of apache to analyse the issue with gdb.

You can manually run the logrotate script in verbose mode like this (-v verbose, -f force):

logrotate -vf /etc/logrotate.d/apache2 

For making a cordump and analyse it with gdb, you can install all the necessary packages using apt-get and it's not very difficult to configure.

Install gdb and necessyry symbol files::

apt-get install gdb 
apt-get install apache2-dbg php5-dbg

Set CoreDumpDirectory in apache config:

CoreDumpDirectory /tmp/apache2-coredump

Set ulimit to unlimited:

ulimit -c unlimited

You can read the coredump with gdb like this:

gdb apache2 -core /tmp/apache2-coredump/core

Here is a detailed how-to: How to figure out what is causing an Apache Segmentation Fault