I'm trying to diagnose a bizarre server crashing issue (Server responds to pings, but won't accept SSH connections until reboot. 0% CPU) where rebooting the server gets everything back to normal. I'd like to have my Apache access logs (or some other log) include all of the requests that had been made right as the crash happened, but unfortunately Apache doesn't log requests until after they complete. Meaning that if a request is crashing the server, that request never finishes and thus doesn't show up in the logs.
Is there some way to configure Apache to create a log file that gets written to as the requests arrive?
Best Answer
I think what you need forensic logging, see this link: http://httpd.apache.org/docs/current/mod/mod_log_forensic.html
snippet:
Forensic Log Format:
Each request is logged two times. The first time is before it's processed further (that is, after receiving the headers). The second log entry is written after the request processing at the same time where normal logging occurs.
In order to identify each request, a unique request ID is assigned. This forensic ID can be cross logged in the normal transfer log using the %{forensic-id}n format string. If you're using mod_unique_id, its generated ID will be used.
The first line logs the forensic ID, the request line and all received headers, separated by pipe characters (|). A sample line looks like the following (all on one line):
+yQtJf8CoAB4AAFNXBIEAAAAA|GET /manual/de/images/down.gif HTTP/1.1|Host:localhost%3a8080|User-Agent:Mozilla/5.0 (X11; U; Linux i686; en-US; rv%3a1.6) Gecko/20040216 Firefox/0.8|Accept:image/png, etc...
The plus character at the beginning indicates that this is the first log line of this request. The second line just contains a minus character and the ID again:
-yQtJf8CoAB4AAFNXBIEAAAAA
The check_forensic script takes as its argument the name of the logfile. It looks for those +/- ID pairs and complains if a request was not completed.