Bash – Apache graceful restart is failing with no errors

amazon-linuxapache-2.4bashelastic-beanstalk

I'm writing a script to gracefully restart Apache in a AWS Elastic Beanstalk application every hour. This is needed so that new SSL certificates are loaded. The app serves multiple websites on multiple domains so each needs its own certificates and config file. If there are no new files, the restart runs ok and websites are still up. If there are new files (a cron job downloads files from an S3 bucket) the websites go down, and there's nothing in the apache logs as to why it's staying down.

This is the script:

#!/bin/bash
exec 3>&1 1>>/var/log/httpd/cert-sync.log 2>&1
# start
echo "[$(date)]: ----------- Start Sync-----------"

echo "[$(date)]: Syncing certs..."
aws s3 sync s3://s3.bucket/certs /etc/pki/tls/

echo "[$(date)]: Syncing conf..."
aws s3 sync s3://s3.bucket/conf /etc/httpd/conf.d/ssl-vhosts/

echo "[$(date)]: Attempting Apache graceful restart..."
sudo /etc/init.d/httpd graceful

exit 0

This is the result in the log:

[Sat Jul  6 21:24:01 UTC 2019]: ----------- Start Sync-----------
[Sat Jul  6 21:24:01 UTC 2019]: Syncing certs...
[Sat Jul  6 21:24:01 UTC 2019]: Syncing conf...
[Sat Jul  6 21:24:01 UTC 2019]: Attempting Apache graceful restart...
Equivalent Upstart operations: start httpd, stop httpd, restart httpd, status httpd
Gracefully restarting httpd
[OK]
[Sat Jul  6 21:27:01 UTC 2019]: ----------- Start Sync-----------
[Sat Jul  6 21:27:01 UTC 2019]: Syncing certs...
[Sat Jul  6 21:27:01 UTC 2019]: Syncing conf...
[Sat Jul  6 21:27:01 UTC 2019]: Attempting Apache graceful restart...
Equivalent Upstart operations: start httpd, stop httpd, restart httpd, status httpd
Gracefully restarting httpd
/etc/init.d/httpd: line 60: kill: (10478) - No such process
Stopping httpd
stop: Unknown instance: 
Starting httpd
httpd start/running, process 14139
[OK]

It always ends with the message [OK]. There's nothing in /var/log/httpd/error_log. The first time it apparently restarts ok, but it doesn't. The second run of the script doesn't find a process running apache so it tries to start it but it also fails. Restarting app servers from the AWS console doesn't fix it either.

This is running Amazon Linux 2017.03. If I push a new update to the application, the server instance is rebuilt, downloads all files (via an .ebextension that doesn't run in the cron but before the app server is running) and starts apache and everything works ok, so I know it isn't an error in the config files. As it stands, whenever a new certificate is downloaded, the only way to make it work is to push an update to the elastic beanstalk, but I need just an scheduled restart.

I've tried these commands to restart apache:

sudo initctl restart httpd
sudo service httpd restart

I wanted to run httpd configtest before running the graceful restart and loging the result, but I got this:

[Sat Jul  6 21:29:01 UTC 2019]: ----> sudo /etc/init.d/httpd configtest
Equivalent Upstart operations: start httpd, stop httpd, restart httpd, status httpd
Usage:  {start|stop|restart|status|graceful}

Which means it didn't run, and the upstart file isn't configured to run configtest. I haven't found a way to check the config before restarting to see if there's anything different with the files when they're synced with the cron vs with the .ebextension on instance startup.

I also found this in /var/log/messages:

Jul  6 19:53:01 ip-172-31-0-244 init: httpd main process ended, respawning
Jul  6 19:53:01 ip-172-31-0-244 init: httpd main process ended, respawning
Jul  6 19:53:01 ip-172-31-0-244 init: httpd main process ended, respawning
Jul  6 19:53:01 ip-172-31-0-244 init: httpd main process ended, respawning
Jul  6 19:53:01 ip-172-31-0-244 init: httpd main process ended, respawning
Jul  6 19:53:01 ip-172-31-0-244 init: httpd main process ended, respawning
Jul  6 19:53:01 ip-172-31-0-244 init: httpd main process ended, respawning
Jul  6 19:53:01 ip-172-31-0-244 init: httpd main process ended, respawning
Jul  6 19:53:01 ip-172-31-0-244 init: httpd main process ended, respawning
Jul  6 19:53:01 ip-172-31-0-244 init: httpd main process ended, respawning
Jul  6 19:53:02 ip-172-31-0-244 init: httpd respawning too fast, stopped

Perhaps Upstart is trying to respawn httpd before the graceful restart is done? I have no idea what I'm talking about here but there might be something there.

This is the upstart config file:

[Sat Jul  6 21:58:01 UTC 2019]: ----> cat /etc/init/httpd.conf
# apache2 - http server
#
# Apache is a web server that responds to HTTP and HTTPS requests.
# Required-Start:    $local_fs $remote_fs $network $syslog
# Required-Stop:     $local_fs $remote_fs $network $syslog

description "apache2 http server"

start on runlevel [2345]
stop on runlevel [!2345]

# Give up if restart occurs 10 times in 30 seconds.
respawn limit 10 30

script
    . /opt/elasticbeanstalk/support/envvars
    if [ -f /etc/elasticbeanstalk/set-ulimit.sh ]; then
        . /etc/elasticbeanstalk/set-ulimit.sh
    fi
    # Output to standard out for upstart and to the error_log
    /usr/sbin/httpd -D FOREGROUND | tee /var/log/httpd/error_log
end script

respawn

I'm new to upstart so I'm not sure if there is something that could be causing the issue here.

I know this is a lot of info but I'm hoping I've provided enough context. Anyone have an idea what might be going on?

Best Answer

You should rule out a cron issue. Does the script run ok outside of cron, on an interactive bash shell?

A lot of things are stripped out by cron:

Related Topic