Php – Apache too many child process – mpm_event caught SIGTERM shutting down

Apache2ddosmod-securitympm-eventPHP

My Apache with ModSecurity, mod_evasive20 enabled occasionally geting crashed:

These are glimpse of error log:

[core:warn]**: child process 24709 still did not exit, sending a SIGTERM
[core:error]**: child process 24709 still did not exit, sending a SIGKILL
[mpm_event:notice] **: caught SIGTERM, shutting down

Last time Apache a crashed when google analytic reported 2K+ live views. I see too many child process can also be a result of attack like DDoS.
But I want to make sure is that really an attack or a configuration flaw.

Please help.

Machine: Ubuntu 18.04, 16 GB RAM, 8 Core CPU, 180GB SSD

MPM Event Configuration:

<IfModule mpm_event_module>
    StartServers         2
    MinSpareThreads      25
    MaxSpareThreads      150
    ThreadLimit      250
    ThreadsPerChild      250
    MaxRequestWorkers    6250
    ServerLimit      25
    MaxConnectionsPerChild   0
</IfModule>

Detailed Error Log:

[Thu Mar 19 06:25:02.506142 2020] [mpm_event:notice] [pid 1460:tid 139655311739840] AH00489: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.0g Apache configured -- resuming normal operations
[Thu Mar 19 06:25:02.506172 2020] [core:notice] [pid 1460:tid 139655311739840] AH00094: Command line: '/usr/sbin/apache2'
[Thu Mar 19 09:15:42.817591 2020] [core:warn] [pid 1460:tid 139655311739840] AH00045: child process 24709 still did not exit, sending a SIGTERM
[Thu Mar 19 09:15:44.819958 2020] [core:warn] [pid 1460:tid 139655311739840] AH00045: child process 24709 still did not exit, sending a SIGTERM
[Thu Mar 19 09:15:46.822259 2020] [core:warn] [pid 1460:tid 139655311739840] AH00045: child process 24709 still did not exit, sending a SIGTERM
[Thu Mar 19 09:15:48.824514 2020] [core:error] [pid 1460:tid 139655311739840] AH00046: child process 24709 still did not exit, sending a SIGKILL
[Thu Mar 19 09:15:49.826497 2020] [mpm_event:notice] [pid 1460:tid 139655311739840] AH00491: caught SIGTERM, shutting down
[Thu Mar 19 09:15:49.962067 2020] [:notice] [pid 29976:tid 140194146679744] ModSecurity for Apache/2.9.2 (http://www.modsecurity.org/) configured.
[Thu Mar 19 09:15:49.962131 2020] [:notice] [pid 29976:tid 140194146679744] ModSecurity: APR compiled version="1.6.2"; loaded version="1.6.3"
[Thu Mar 19 09:15:49.962136 2020] [:warn] [pid 29976:tid 140194146679744] ModSecurity: Loaded APR do not match with compiled!
[Thu Mar 19 09:15:49.962140 2020] [:notice] [pid 29976:tid 140194146679744] ModSecurity: PCRE compiled version="8.39 "; loaded version="8.39 2016-06-14"
[Thu Mar 19 09:15:49.962144 2020] [:notice] [pid 29976:tid 140194146679744] ModSecurity: LUA compiled version="Lua 5.1"
[Thu Mar 19 09:15:49.962147 2020] [:notice] [pid 29976:tid 140194146679744] ModSecurity: YAJL compiled version="2.1.0"
[Thu Mar 19 09:15:49.962150 2020] [:notice] [pid 29976:tid 140194146679744] ModSecurity: LIBXML compiled version="2.9.4"
[Thu Mar 19 09:15:49.962154 2020] [:notice] [pid 29976:tid 140194146679744] ModSecurity: Original server signature: Apache/2.4.29 (Ubuntu)
[Thu Mar 19 09:15:49.962157 2020] [:notice] [pid 29976:tid 140194146679744] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Thu Mar 19 09:15:50.025004 2020] [mpm_event:notice] [pid 29986:tid 140194146679744] AH00489: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.0g Apache configured -- resuming normal operations
[Thu Mar 19 09:15:50.025066 2020] [core:notice] [pid 29986:tid 140194146679744] AH00094: Command line: '/usr/sbin/apache2'

If any additional info need, please mention. Thanks.

Best Answer

Many factors can lead to Apache restarting or shutting down. To dig a bit deeper, you can run these commands then follow the bread crumbs.

cat /var/log/messages | grep apache

cat /var/log/syslog | grep apache

cat /var/log/messages | grep kill

If you have multiple sites running (sites-available) with their own error logs, it would also be prudent to check each one's error log to ensure help narrow down any issues. This is necessary because Apache could be doing a graceful restart, and upon restarting, a site config could be causing a fatal error that is will not show in system logs.

If they are not already, make sure error logging is on for all sites.

Also, although obvious, make sure you are updated. Plugin version issues can often cause Apache to crash or fail to restart.

sudo apt-get update && sudo apt-get upgrade -y

Always check your global and site configs with this command. Especially 000-default.conf, apache2.conf, and ports.conf.

sudo apachectl configtest

Ensure that this command does not return an error as well.

apache2ctl restart

If you have the patience and the ability, you can also try disabling mod one at a time. Start with ones like mod_fastcgi, rewrite, and anything SSL related as they are common culprits.

SIGTERM is used to restart Apache and is worth reading up on if you get this far without a solution (assuming its setup in init to auto-restart): http://httpd.apache.org/docs/2.2/stopping.html

As a last resort, a clean install of Apache and PHP will likely clear out the issue. I highly recommend using docker / docker-compose to run your stack if you can. It makes keeping your apache, PHP, and database updated a very painless process.

Related Topic