I'm draining various logs from Heroku to an rsyslog linux (ubuntu) server and am starting to have a little more to bite off than I can chew in terms of working with my log histories.
I am needing to be able drill back in time based on more flexible details and more flexible access than what the standard syslog file(s) provide.
I'm thinking that logging to mysql may be the correct approach, but how do I set this up such that it pulls only certain log entries into a table based on an identified? For example, I see a long hex string identifying each log entry from a certain Heroku app instance. I assume that I can just pipe those into the mysql socket vs ALL rsyslog input into mysql…
Could someone please direct me to a resource that can walk me through the process of setting something like this up or simply provide some details that can help? I have 15+ years of Unix experience so I just need some nudging in the right direction as I've not really done a tremendous amount of work with syslog daemons previously in terms of pooling various servers into one.
Additionally, I'd be interested in any log review tools that could make drilling through log arrangements like this more handy for developers.
Best Answer
I guess something along this should work. First, make sure you have the line below in your rsyslog conf file, so MySQL output module gets loaded:
Then, filter the stuff you want like this:
Next you need to create a template for your data, so rsyslog knows what and how to insert to your database. Here is an example template which only records date and message, and is excepting you to have a table with only two columns,
dateandmessage.Something like this should work. Note that this was on the top of my head and not tested at all, but you get the idea. Much more detailed information can be found from rsyslog web site, for example the available template parameter values might interest you.