I have several servers running Debian Stretch and I keep running into an issue where the process will hit a segmentation fault and stop. It will not return to service until I manually run service apache2 restart
. I'm trying to figure out what is causing it so I can keep the server up and running but I have been unable to.
The server is running two WordPress instances (one a public site, the other a private staging site for content purposes). Both are protected by Let's Encrypt via Certbot (I include this because of the [ssl:warn]
in the error log below). We have not observed any memory or disk space issues when this has happened. Swap on these servers is hardly ever used.
Here is the output for service apache2 status
after a segfault:
# service apache2 status
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Sun 2018-07-08 15:50:24 MST; 29min ago
Process: 11833 ExecStop=/usr/sbin/apachectl stop (code=exited, status=0/SUCCESS)
Process: 11828 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
Main PID: 883 (code=exited, status=0/SUCCESS)
Jul 08 15:50:24 hostname systemd[1]: Starting The Apache HTTP Server...
Jul 08 15:50:24 hostname apachectl[11828]: httpd (pid 11770) already running
Jul 08 15:50:24 hostname systemd[1]: Started The Apache HTTP Server.
Here is the output in /var/log/apache2/error.log
:
[Sat Jul 07 17:04:51.693795 2018] [core:notice] [pid 29385] AH00052: child pid 18866 exit signal Segmentation fault (11)
[Sat Jul 07 17:04:51.693918 2018] [mpm_prefork:notice] [pid 29385] AH00169: caught SIGTERM, shutting down
[Sat Jul 07 17:04:52.484310 2018] [ssl:warn] [pid 19421] AH01906: bb7f602e547898d78a02b844d49c34bc.4210997990497fe5b452e5c6c4250620.acme.invalid:443:0 server certificate is a C
A certificate (BasicConstraints: CA == TRUE !?)
/page/8/
[Sat Jul 07 17:04:51.693795 2018] [core:notice] [pid 29385] AH00052: child pid 18866 exit signal Segmentation fault (11)
[Sat Jul 07 17:04:51.693918 2018] [mpm_prefork:notice] [pid 29385] AH00169: caught SIGTERM, shutting down
[Sat Jul 07 17:04:52.484310 2018] [ssl:warn] [pid 19421] AH01906: bb7f602e547898d78a02b844d49c34bc.4210997990497fe5b452e5c6c4250620.acme.invalid:443:0 server certificate is a C
A certificate (BasicConstraints: CA == TRUE !?)
[Sat Jul 07 17:04:52.495766 2018] [ssl:warn] [pid 19422] AH01906: bb7f602e547898d78a02b844d49c34bc.4210997990497fe5b452e5c6c4250620.acme.invalid:443:0 server certificate is a C
A certificate (BasicConstraints: CA == TRUE !?)
[Sat Jul 07 17:04:52.498208 2018] [mpm_prefork:notice] [pid 19422] AH00163: Apache/2.4.25 (Debian) OpenSSL/1.0.2l configured -- resuming normal operations
[Sat Jul 07 17:04:52.498230 2018] [core:notice] [pid 19422] AH00094: Command line: '/usr/sbin/apache2'
[Sat Jul 07 17:04:58.754662 2018] [mpm_prefork:notice] [pid 19422] AH00171: Graceful restart requested, doing restart
[Sat Jul 07 17:04:58.766272 2018] [mpm_prefork:notice] [pid 19422] AH00163: Apache/2.4.25 (Debian) OpenSSL/1.0.2l configured -- resuming normal operations
[Sat Jul 07 17:04:58.766290 2018] [core:notice] [pid 19422] AH00094: Command line: '/usr/sbin/apache2'
[Sat Jul 07 17:05:00.039384 2018] [mpm_prefork:notice] [pid 19422] AH00171: Graceful restart requested, doing restart
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
[Sat Jul 07 17:05:00.050665 2018] [ssl:warn] [pid 19422] AH01906: 2af61f923209309052c60f342e6a0578.4287ae6d0b1c48707d1262e562b6250a.acme.invalid:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Sat Jul 07 17:05:00.051519 2018] [mpm_prefork:notice] [pid 19422] AH00163: Apache/2.4.25 (Debian) OpenSSL/1.0.2l configured -- resuming normal operations
[Sat Jul 07 17:05:00.051528 2018] [core:notice] [pid 19422] AH00094: Command line: '/usr/sbin/apache2'
[Sat Jul 07 17:05:06.063638 2018] [core:error] [pid 19422] AH00546: no record of generation 0 of exiting child 19423
[Sat Jul 07 17:05:06.420374 2018] [mpm_prefork:notice] [pid 19422] AH00171: Graceful restart requested, doing restart
[Sat Jul 07 17:05:06.431243 2018] [mpm_prefork:notice] [pid 19422] AH00163: Apache/2.4.25 (Debian) OpenSSL/1.0.2l configured -- resuming normal operations
[Sat Jul 07 17:05:06.431264 2018] [core:notice] [pid 19422] AH00094: Command line: '/usr/sbin/apache2'
[Sat Jul 07 17:05:07.965690 2018] [mpm_prefork:notice] [pid 19422] AH00171: Graceful restart requested, doing restart
[Sat Jul 07 17:05:07.976624 2018] [mpm_prefork:notice] [pid 19422] AH00163: Apache/2.4.25 (Debian) OpenSSL/1.0.2l configured -- resuming normal operations
[Sat Jul 07 17:05:07.976636 2018] [core:notice] [pid 19422] AH00094: Command line: '/usr/sbin/apache2'
[Sat Jul 07 17:05:07.977526 2018] [core:error] [pid 19422] AH00546: no record of generation 0 of exiting child 19550
[Sat Jul 07 17:05:08.211152 2018] [core:notice] [pid 19422] AH00052: child pid 19531 exit signal Segmentation fault (11)
[Sat Jul 07 17:05:08.211291 2018] [mpm_prefork:notice] [pid 19422] AH00169: caught SIGTERM, shutting down
We have the following software and hardware for the logs above (I can provide anything else that may be helpful):
- apache2 2.4.25-3+deb9u4
- Debian Stretch 9.4
- PHP 7.0.27-0+deb9u1 with FPM
- mariadb 10.1.26-0+deb9u1
- 4x cores of Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
- 8G RAM
- 512MB swap
- 95G SSD
Best Answer
The issue was caused by Certbot trying to renew certificates. If I ran
certbot renew
, I ran into these errors (I've sanitized the log a bit to remove domains and IPs):Afterward, running
service apache2 status
resulted inI enabled the vhost in question, restarted Apache, and re-ran
certbot renew
and everything worked just fine. It was crashing frequently because certbot tries to renew twice daily by default.