I guess something along this should work. First, make sure you have the line below in your rsyslog conf file, so MySQL output module gets loaded:
$ModLoad ommysql
Then, filter the stuff you want like this:
:msg, contains, "Heroku" :ommysql:dbserver,dbname,dbuser,dbpass;heroku-template
Next you need to create a template for your data, so rsyslog knows what and how to insert to your database. Here is an example template which only records date and message, and is excepting you to have a table with only two columns, date
and message
.
$template heroku-template, "INSERT INTO herokulog(date,message) VALUES ('%timestamp','%msg')\r\n", SQL
Something like this should work. Note that this was on the top of my head and not tested at all, but you get the idea. Much more detailed information can be found from rsyslog web site, for example the available template parameter values might interest you.
The problem was actually coming from logrotate.
Basically with my configuration, running unicorn, I don't need to use the copytruncate
directive. (which is what causes problems here)
USR1 - Reopen all logs owned by the worker process. See
Unicorn::Util.reopen_logs for what is considered a log. Log files are
not reopened until it is done processing the current request, so
multiple log lines for one request (as done by Rails) will not be
split across multiple logs.
This started working properly after updating to this configuration:
/home/user/my_app/shared/log/*.log {
daily
missingok
dateext
rotate 30
compress
notifempty
extension gz
create 640 user user
sharedscripts
post-rotate
# Telling Unicorn to reload files
test -s /home/user/my_app/shared/pids/unicorn.pid && kill -USR1 "$(cat /home/user/my_app/shared/pids/unicorn.pid)"
# Reloading rsyslog telling it that files have been rotated
reload rsyslog 2>&1 || true
endscript
}
Best Answer
Try this LEGACY rsyslog formatted version:
You can do similar entries for your other log files.
After that, create some extractors on your graylog2 server for the 12514/TCP input. This will give you some fine grain options for graphs etc.