Fixing severity on graylog2 web interface

grayloglogstash

I am using logstash to collect logs from a group of webapps and send them to graylog2 for centralized viewing.

I have the following filter for tokenizing:

 grok {
   type => "webapps"
   pattern => "^%{TIME:timestamp} \[%{NOTSPACE:thread}\]%{SPACE}%{WORD:loglevel}%{SPACE}%{JAVACLASS:class} - %{GREEDYDATA:short_message}"
   drop => false
   debug => "true"
   add_tag => [ "%{loglevel}" ]
 }

Loglevel will be along the lines of TRACE, DEBUG, INFO, NOTICE, ERROR, FATAL.

Initially, everything was showing up as "Alert" (which has a numeric value of 5 in ruby/graylog-server and is marked in the sources as "unknown").

I then added a series of mutate filters, such as this ones:

 mutate {
   type => "webapps"
   tags => "INFO"
   add_tag => [ "ll_%{@level}", "mutated" ]
   replace => [ "@level", "6" ] # informational
 }

This got me closer to the proper loglevel/severity showing up in the webinterface, but all the "INFO" messages were showing up as being Debug messages.

I wrote a script to look through the elasticsearch data and set the level field based on what it should be.

  • Debug : 7
  • Informational : 6
  • Warning : 4
  • Error : 3
  • Critical : 2

with levels 0, 1 and 5 not being used, as the sources indicated these were reserved.

But this script is expensive in terms of resources, and I don't think it would work too well when the dataset it is trying to update is growing at a rate of hundreds or thousands of messages per second.

The other thing I noticed is that when I use something like "@source" in a tag, the proper value shows up. When I do "@level", I get that inserted as a literal, which indicates that it isn't a predefined field has had been indicated.

I've looked at the sources a little, but not exhaustively, so it's possible I've just missed it.

The question is, what do I need to change in my mutate filter to make "INFO" show up as "Informational" in the graylog2 interface?

Best Answer

Nothing I tried worked, and the mailing list wasn't able to help either.

What I wound up doing was to stop using logstash for this, and add in a logback-gelf jar and add to my logback configuration.

It has worked like a charm since then.