we have several (usually java) application and they log to centralized rsyslog server (version 7.6). We had problems with java exceptions as rsyslog was handling each line of stack trace as separate message, so our developers started to use #012 as newline and #011 as tab. So now, when I have message with exception, it looks like:
2015-02-09T12:52:43.769Z hostname warning app Exception when invoking method something.search: reason=Error while executing method something.search, Params=#012{"maxResults":10,"query":"somesearch"}#012com.company.server.json.ApiException: Error while executing method something.search#012#011at com.company.app.api.BaseExecutor.executeWithDatabase(BaseExecutor.java:112) ~[app-server.jar:?]#012#011at com.company.app.api.BaseExecutor.execute(BaseExecutor.java:87) ~[app-server.jar:?]#012#011at com....and so on
The first thing I would love to is to automatically replace those #xxx strings for their original values, so the resulting logfile is readable.
The second thing is, we are using graylog2 where we temporarily for a few weeks store the logs so it can be easily searched. Rsyslog is setup to save the logs to some location AND resend the logs to graylog and I of course would like to have exceptions there with spaces and tabulators instead of control characters.
I was doing some research and was trying to replace these control characters as they should (I think) be compatible with rsyslog:
http://www.rsyslog.com/doc/property_replacer.html
http://www.rsyslog.com/doc/rsconf1_escapecontrolcharactersonreceive.html
But no luck so far.
Thank you in advance for your time and help!
P.S. First issue can be used with cron and sed, but it wont solve the second one.
Best Answer
Not exactly a solution, but good enough workaround.
For rsyslog, yes, nightly cron job with a little 'sed' command For graylog: You can use drools rules to rewrite the message before it reaches graylog engine. (source: https://www.graylog.org/documentation/general/rewriting/ )