Ubuntu – Sending logs to Graylog2 server

apache-2.2graylogloggingrsyslogUbuntu

I have just setup a Graylog2 server and I am looking to send all logs from my main server to the graylog server. I have enabled logging for the main server and am sending logs to my graylog server by adding *.* @logs.example.com:1337 to /etc/rsyslog.conf.

What I want is to have Graylog2 collect all my Apache logs, system logs (for SSH logins, rejected logins) and any other logs I need to monitor.

For the Apache logs, I would also like the Rails logs. My sites are located in /srv/www/ and then the structure is sitename.com/public_html and sitename.com/logs. I have many sites on the server and I would like an easy way to view all of the errors and make some nice graphs out of them hence why I want to use Graylog2…

The log files in the logs folder are access.log and error.log.

The Rails logs would be in sitename.com/public_html/log. This contains production.log.

Best Answer

This is old, but I thought I would write this method which I use for low/medium traffic site (don't know if it will work well for heavy traffic site):

In Apache, I define a CustomLog format called graylog2_access which formats the access log into a GELF format and then I send my log through Graylog2 by piping the log data through nc to send GELF messages to Graylog2's input.

Here is the custom format that it creates (human readable):

{ 
 "version": "1.1",
 "host": "%V",
 "short_message": "%r",
 "timestamp": %{%s}t,
 "level": 6,
 "_user_agent": "%{User-Agent}i",
 "_source_ip": "%a",
 "_duration_usec": %D,
 "_duration_sec": %T,
 "_request_size_byte": %O,
 "_http_status": %s,
 "_http_request_path": "%U",
 "_http_request": "%U%q",
 "_http_method": "%m",
 "_http_referer": "%{Referer}i"
}

For the Apache config, here is a copy/paste version:

LogFormat "{ \"version\": \"1.1\", \"host\": \"%V\", \"short_message\": \"%r\", \"timestamp\": %{%s}t, \"level\": 6, \"_user_agent\": \"%{User-Agent}i\", \"_source_ip\": \"%a\", \"_duration_usec\": %D, \"_duration_sec\": %T, \"_request_size_byte\": %O, \"_http_status\": %s, \"_http_request_path\": \"%U\", \"_http_request\": \"%U%q\", \"_http_method\": \"%m\", \"_http_referer\": \"%{Referer}i\" }" graylog2_access

Then in your host configuration:

CustomLog "|nc -u graylogserver 12201" graylog2_access
Related Topic