Debian – How to automatically re-start apache if it stops responding and socket is still in use

apache-2.2debiandebian-squeezekill-processmono

Apache server Debian Linux x64 serves the site mono applications in mysite.com and other sites using virtual hosting.

Sometimes apache stops responding for unknow reason. Apache error_log contains:

[Sat Jun 20 13:56:31 2015] [error] [client 90.190.33.164] File does not exist: /var/www/apple-touch-
icon.png
[Sat Jun 20 16:38:04 2015] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Sat Jun 20 17:20:11 2015] [notice] caught SIGTERM, shutting down
mod-mono-server received a shutdown message
[Tue Jun 23 09:20:50 2015] [error] mod_mono: connect error (Interrupted system call). File: /tmp/mod
_mono_server_default
mod-mono-server received a shutdown message
mod-mono-server received a shutdown message
mod-mono-server received a shutdown message
[Tue Jun 23 09:20:54 2015] [error] Not running mod-mono-server.exe because no MonoApplications, Mono
ApplicationsConfigFile or MonoApplicationConfigDir specified.
[Tue Jun 23 09:20:54 2015] [notice] Apache/2.2.16 (Debian) mod_mono/2.6.3 configured -- resuming normal operations
..

It looks like

[Sat Jun 20 16:38:04 2015] [error] server reached MaxClients setting, consider raising the MaxClients setting

message in log file appears first.

After that probably cron job tries below to re-start apache which causes message in error_log:

[Sat Jun 20 17:20:11 2015] [notice] caught SIGTERM, shutting down
mod-mono-server received a shutdown message

Server was down starting at this time: access_log does not log any access starting from Jun 20 17:20

Server starts resoping only at [Tue Jun 23 09:20:50 2015] when I manually re-started it.
This start to occur from last month several times per month.

To fix this the following crontab script is used running after every 10 minutes:

wget --timeout=10 --no-verbose --tries=1 -a /var/log/wget.log -O /var/log/wgettulem.html  mysite.com || /etc/init.d/apache2 restart

If apache stops responding, log file written by this script contains

Connection timed out error messages.

However apache is not restarted.
Trying to re-start it manually using

/etc/init.d/apache2 restart

returns Sockect is in use error. ps aux shows that apace is stil running.

I fix this by using

ps aux | grpe apache

to find appache process number and use

kill

to kill this process.

How to fix or find the reson of this message ?
How to automate this process so that all apache instances are killed and apache will be re-started automatically by this script. How is there bettor way to keep apache responding ?

Debian version is 6.0.4

apache2 -V returns

Server version: Apache/2.2.16 (Debian)
Server built:   Feb  5 2012 21:35:42
Server's Module Magic Number: 20051115:24
Server loaded:  APR 1.4.2, APR-Util 1.3.9
Compiled using: APR 1.4.2, APR-Util 1.3.9
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/etc/apache2"
 -D SUEXEC_BIN="/usr/lib/apache2/suexec"
 -D DEFAULT_PIDLOG="/var/run/apache2.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="/var/run/apache2/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="mime.types"
 -D SERVER_CONFIG_FILE="apache2.conf"

apache2.conf contains

<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

Best Answer

If you merely want to forcibly restart Apache if it hangs (for any reason) then you should install System Integrity Monitor by R-fx Networks.

Among many features, it can be configured to monitor your web services, automatically issue restart commands and send out notifications.

Quick install instructions:

cd /usr/local/src
wget http://www.rfxn.com/downloads/sim-current.tar.gz
tar -xvf sim-current.tar.gz
cd sim*
./setup -i

Proceed with the interactive installation defining the appropriate filepaths for your web service executables. As a last step, enable the cronjob:

/usr/local/sbin/sim -j

Related Solutions

Linux – Apache server keeps crashing regularly

I'm not saying this is what's happening but based on my own experience as a CentOS admin, it's most likely runaway apache/php processes taking down the server. I've seen this numerous times on CentOS 5. It's frustrating because there's usually not a trace of what happened in the log files. The machine just grinds to a halt due to physical memory and swap being sucked up by apache/php processes. You would think linux memory management or some daemon would jump in and say "hey stop" but it doesn't. It'll let apache grind your system to a halt.

Having said that, to see what's happening you'll need something that can monitor and log resource usage. I like to use a program called atop. Atop is a lot like the top program but it also takes a snapshot of resource usage at defined intervals. It's pretty simple to install.

wget http://www.atcomputing.nl/Tools/atop/packages/atop-1.23.tar.gz 
tar -zxvf atop-1.23.tar.gz
cd atop-1.23 && make install

Open /etc/atop/atop.daily with a text editor and change INTERVAL=600 to INTERVAL=60

Run the command /etc/atop/atop.daily from a command prompt to start it. Wait a few minutes and run atop -r /var/log/atop/atop_20091118 with the correct date of course.

Hit the t key to go forward in time and T to go back. Next time your server crashes do this and check the MEM free and SWP free lines. If you have memory problems these will be in red. Also look for numerous httpd lines under CMD. If apache/php is your problem there'll be a bunch of them.

If this is the case, I recommend looking at you're MaxClients setting in httpd.conf. If set too high, apache will gladly eat all of your memory causing your machine to crash. Apache/php can easily eat 40-50MB/process. If you multiply 40mb x MaxClients you'll get a rough idea of how much memory apache can potentially use. MaxClients usually defaults to 150 on CentOS so apache can potentially use 6GB of memory by default. This doesn't include memory your system needs for itself and other processes to run. Try setting it to a more realistic value based on the amount of memory you have like 40 if you have 2G of memory and see if that helps. Also if you have KeepAlive On, set KeepAliveTimeout to a low number like 2 or 3.

In my opinion CentOS's apache/php compilation is a real pos that should never have seen the light of day. It's buggy and crash prone. If you run a serious site, I highly recommend compiling your own version of apache/php or even using one of the newer high performance webservers like lighttpd or nginx with fgci php.

Php writing deprecated errors to apache error log

While the other answers here will stop the errors being written to your error log, they are simply ignoring the the error message and not fixing the error.

The error in this case is that your php.ini still has either magic_quotes_gpc on or magic_quotes_gpc off somewhere in it. The same is true for register_globals on or register_globals off.

The error is not that the directive is on or off. The error is that the directive shouldn't exist at all. Comment those lines out of your php.ini or remove them entirely and PHP will stop writing errors about deprecated directives.

Of course, this may cause problems with your application if it requires either of those to be on.

The reason this is an error in PHP 5.3 is that in PHP 6, these directives won't even exist and PHP 6 will behave as if they were set to off. If you ever plan on upgrading to PHP 6, now is a good time to start upgrading or replacing your application.

Another solution you could try is downgrading PHP back to the 5.2 or 5.1 branch.

As for PHP writing errors into Apache's log, this is natural because PHP is running as an Apache module. You can put something like error_log = /var/log/php_errors.log into your php.ini and restart Apache to have the PHP errors separated from your Apache errors. While you're in there, I would recommend changing display_errors to off. Error messages can often contain sensitive information that you wouldn't want an attacker to see. You will most likely see this written in your php.ini:

; - display_errors = Off           [Security]
;     With this directive set to off, errors that occur during the execution of
;     scripts will no longer be displayed as a part of the script output, and thus,
;     will no longer be exposed to remote users.  With some errors, the error message
;     content may expose information about your script, web server, or database
;     server that may be exploitable for hacking.  Production sites should have this
;     directive set to off.

There is no sane reason why the error messages contain HTML.

To answer another question that you didn't ask, the reason PHP reports this as being in <b>Unknown</b> on line <b>0</b> is that the error message was designed for lines of PHP code that you have written but the error it found was in parsing the php.ini before it has even read a single line of code or even opened a .php file. Since it hasn't opened a file and doesn't have a line number, it reports them as "Unknown" and "0".

Best Answer

Related Solutions

Linux – Apache server keeps crashing regularly

Php writing deprecated errors to apache error log

Related Topic