Apache Piping Log to Netcat Fails – Troubleshooting Guide

apache-2.2graylognetcat

I want to send my Apache log in a custom format (GELF) to a UDP receiving server (running Graylog2). I was sure everything was working fine but then after a little while, I got an alert, my server was not responding. I see in the Apache error log a whole bunch of:

piped log program 'nc -w 1 -u logserver 12201' failed unexpectedly

Interestingly, on a server with a very minimal traffic, I see this error constantly in the logs even when there is no traffic to the web site. And when there is traffic, the log does get sent by netcat to graylog – so it does work regardless.

If I stop Apache and restart it, the error continues to show up in error log after restart. I made sure after stopping it that there were not runaway processes left over.

The configuration is:

LogFormat "{ \"version\": \"1.1\", \"host\": \"%V\", \"short_message\": \"%r\", \"full_message\": \"%r, status: %>s, %O bytes, User Agent: %{User-Agent}i\", \"timestamp\": %{%s}t, \"level\": 6, \"_user_agent\": \"%{User-Agent}i\", \"_source_ip\": \"%a\", \"_duration_usec\": %D, \"_duration_sec\": %T, \"_request_size_byte\": %O, \"_http_status\": %s, \"_http_request_path\": \"%U\", \"_http_request\": \"%U%q\", \"_http_method\": \"%m\", \"_http_referer\": \"%{Referer}i\" }" graylog2_access

And the CustomLog:

CustomLog "|nc -w 1 -u logserver 12201" graylog2_access

I'm not sure how to get more debug information as to what failed.

  1. Can someone help me get more detailed info on the failure?
  2. If someone knows why this fails, then that would be a great answer too!
  3. Another way to ship the logs constantly (realtime) is an acceptable solution too. However, I know about logstash but in this case, I don't need parsing, I can already output in GELF format. I tried logstash in the past too and it was always eventually running out of memory and stopping on its own.

Best Answer

Apache docs: http://httpd.apache.org/docs/2.2/logs.html#piped

Apache will start the piped-log process when the server starts, and will restart it if it crashes while the server is running. (This last feature is why we can refer to this technique as "reliable piped logging".)

NC docs: man nc

 -w timeout
         Connections which cannot be established or are idle timeout
         after timeout seconds.  The -w flag has no effect on the -l
         option, i.e. nc will listen forever for a connection, with
         or without the -w flag.  The default is no timeout.

What you are seeing is Apache starting netcat with a timeout of 1 second. Regardless of if there is any log data, netcat then times out after one second (due to -w 1 option) and quits. Apache then restarts netcat. Lather, rinse, repeat.

I suggest removing -w 1 from the netcat command in this circumstance.

Stepping back, I would suggest using syslog for this function (not sure why you didn't go with it in the first place).

Related Topic