I've had my own virtual machine based server running at a professional hosting company for half a year. I have not made any changes in months.
Suddenly, a few days ago, my 3rd party site monitoring system emailed me that all my sites are down. I restarted apache, and it worked again, but the next day the same thing happened again. Now I'm on my 3rd day in a row of this. At least the last two days, it happened between 7am and 8am my time.
All I get in the Apache error log (/var/log/apache2/error_log) is this:
[Mon May 27 00:33:17.764703 2019] [core:notice] [pid 5405] AH00094: Command line: '/usr/sbin/apache2'
[Mon May 27 00:33:17.927290 2019] [mpm_prefork:notice] [pid 5405] AH00169: caught SIGTERM, shutting down
[Mon May 27 00:52:55.866838 2019] [mpm_prefork:notice] [pid 848] AH00163: Apache/2.4.25 (Debian) OpenSSL/1.0.2r configured -- resuming normal operations
[Mon May 27 00:52:55.879748 2019] [core:notice] [pid 848] AH00094: Command line: '/usr/sbin/apache2'
[Mon May 27 12:47:22.759575 2019] [mpm_prefork:notice] [pid 848] AH00169: caught SIGTERM, shutting down
[Mon May 27 12:47:28.041888 2019] [mpm_prefork:notice] [pid 5572] AH00163: Apache/2.4.25 (Debian) OpenSSL/1.0.2r configured -- resuming normal operations
[Mon May 27 12:47:28.041936 2019] [core:notice] [pid 5572] AH00094: Command line: '/usr/sbin/apache2'
[Mon May 27 12:47:28.212269 2019] [mpm_prefork:notice] [pid 5572] AH00169: caught SIGTERM, shutting down
[Mon May 27 15:54:59.721916 2019] [mpm_prefork:notice] [pid 6550] AH00163: Apache/2.4.25 (Debian) OpenSSL/1.0.2r configured -- resuming normal operations
[Mon May 27 15:54:59.721963 2019] [core:notice] [pid 6550] AH00094: Command line: '/usr/sbin/apache2'
It is the 3rd line from the bottom when the system went down, I believe. It looks like apache shut down but did not start up again.
I know enough about Linux to be dangerous, but I'm running out of ideas of where else to look.
What other log files would you recommend checking, and what other troubleshooting steps can I take, e.g. add extra logging, etc.?
This is running on Debian 9, and no one else has access to anything on the server. It runs Apache2, MySQL and PHP7.
Thanks!
Best Answer
SIGTERM sent to httpd means stop now. Find what is sending this signal.
killsnoop is an easy script to do this. However, BPF won't be available on Debian 9. (Sometimes digging in the bcc repo reveals other pre-BPF implmentations, but not always.)
Or, you can setup auditd to log kill syscalls.
Anti-patterns to look for, in crontabs and anywhere else: