On my Debian machine (running the apache as packaged by Debian, package 2.4.7-1), every time apache restarts through apache2ctl restart or apache2ctl graceful, it forgets about my cgi-bin directory:
me@aram:~$ sudo apache2ctl graceful-stop
me@aram:~$ sudo apache2ctl graceful
httpd not running, trying to start
me@aram:~$ curl -s http://localhost/cgi-bin/hello
Hello world, from a cgi script.
me@aram:~$ sudo apache2ctl graceful
me@aram:~$ curl -s http://localhost/cgi-bin/hello
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /cgi-bin/hello was not found on this server.</p>
<hr>
<address>Apache/2.4.7 (Debian) Server at localhost Port 80</address>
</body></html>
me@aram:~$ sudo apache2ctl graceful
me@aram:~$ curl -s http://localhost/cgi-bin/hello
Hello world, from a cgi script.
me@aram:~$ sudo apache2ctl graceful; sudo apache2ctl graceful
me@aram:~$ curl -s http://localhost/cgi-bin/hello
Hello world, from a cgi script.
me@aram:~$ sudo apache2ctl graceful
me@aram:~$ curl -s http://localhost/cgi-bin/hello
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /cgi-bin/hello was not found on this server.</p>
<hr>
<address>Apache/2.4.7 (Debian) Server at localhost Port 80</address>
</body></html>
me@aram:~$ sudo apache2ctl graceful
me@aram:~$ curl -s http://localhost/cgi-bin/hello
Hello world, from a cgi script.
This is perfectly repeatable; if I stop the server completely and restart it, it's fine. After that, a restart (graceful or not) switches behaviors. Unfortunately, the regular logrotate job issues a graceful restart, so I need to make it so that Apache always knows about cgi-bin.
There's nothing in the access or error log to mark a "will know about cgi" restart from a "won't know about cgi" restart. If people comment with suggestions as to how to make the logs more verbose, I'll try that and update this.
I'm using the config that the Debian package comes with (so it's configured by symlinking a bunch of files in -enabled directories to existing files in -available directories). Here, for reference, are what my -enabled directories contain:
me@aram:/etc/apache2$ ls *-enabled
conf-enabled:
000-local-routerblock.conf charset.conf javascript-common.conf other-vhosts-access-log.conf serve-cgi-bin.conf
apache2-doc.conf dwww.conf localized-error-pages.conf security.conf
mods-enabled:
access_compat.load authn_core.load authz_host.load cgi.load dir.load filter.load mpm_prefork.load reqtimeout.load status.load
alias.conf authn_file.load authz_user.load deflate.conf dnssd.conf mime.conf negotiation.conf setenvif.conf userdir.conf
alias.load authz_core.load autoindex.conf deflate.load dnssd.load mime.load negotiation.load setenvif.load userdir.load
auth_basic.load authz_groupfile.load autoindex.load dir.conf env.load mpm_prefork.conf reqtimeout.conf status.conf
sites-enabled:
000-default.conf
All of those, except for 000-local-routerblock.conf, are as shipped by Debian. I've verified that removing that config file fixes the issue, so I suspect it's some odd interaction between that file and the default serve-cgi-bin.conf file. Again, though, no evidence in the logs.
Contents of 000-local-routerblock.conf:
Redirect /router /cgi-bin/router
<Location "/cgi-bin/router">
AuthUserFile /usr/lib/cgi-bin/routerblock/htpasswd
AuthGroupFile /dev/null
AuthName ByPassword
AuthType Basic
Require user router
</Location>
Contents of serve-cgi-bin.conf:
<IfModule mod_alias.c>
<IfModule mod_cgi.c>
Define ENABLE_USR_LIB_CGI_BIN
</IfModule>
<IfModule mod_cgid.c>
Define ENABLE_USR_LIB_CGI_BIN
</IfModule>
<IfDefine ENABLE_USR_LIB_CGI_BIN>
ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Require all granted
</Directory>
</IfDefine>
</IfModule>
# vim: syntax=apache ts=4 sw=4 sts=4 sr noet
EDIT: I've solved this now, in terms of "I made it work". I have no idea why this works, and that it does almost certainly indicates a bug in this version of Apache. I'm leaving the bounty up for someone to answer as to what is/was going on.
In any case, the solution was not in any of the config. files I quoted. Remember, the rest of this configuration is exactly stock Debian, nothing custom about the configuration. The Debian /etc/apache2/apache2.conf includes the line:
LogLevel warn
In the stock configuration, there are no other LogLevel statements, except for a commented out one in the file /etc/apache2/sites-available/000-default.conf. (that entire file is inside a <VirtualHost *:80> tag; it sets up DocumentRoot and the access log, but nothing else)
I was messing around with LogLevel settings trying to get more information in my logs when I stumbled across this: if I have a LogLevel statement inside that VirtualHost tag, it works. So now in addition to the LogLevel warn that's in the main apache config, I also have a LogLevel warn inside the <VirtualHost *:80> tag in 000-default.conf. This works. If I have extra time this weekend, I think I'll try to set up a virtual machine to reproduce this in as clean an environment as possible, and then file a Debian bug.
Best Answer
Based on the comments and link to a bug report above, this does appear to have been a valid bug that has been recently resolved; from https://bugs.debian.org/743860 :