Graylog: Fields from Pipeline rule not showing up in search data

graylog

I'm trying out Graylog for system logs and Snort alerts. I've followed the example here to get my snort alerts into Graylog and then proceeded to add another Stream, Pipeline and Rule for a separate IDS log source. I basically copied the Snort example and changed the Regex to extract the pertinent fields from the new log source. The new Regex does test OK when I select a message from the search tab and then select "Test against stream". Messages are also showing up under my new stream when selected from the Streams menu item at the top of Graylog.

The thing is, in the Pipeline rule, I use the set_field() function to assign the value of the matching group from the regex. I've got nine fields in the Rule using set_field() however none of them are showing up in the search data. Why is this? Do I need to use add_field() first like in the GELF examples? I was assuming set_field() automatically did that as that is how the snort example at the link above works. Attached my Pipeline rule below which is attached to my IDrops stream in Graylog.

rule "Extract IDropS fields"
when
  has_field("message")
then
  let m = regex("^([a-z]+)\\s.*(TCP|UDP|ICMP)\\s([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}):(\\d{1,5})\\s([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}):(\\d{1,5}).*SnortSid.*:([0-9]+):([0-9]+)\\s(.*)$", to_string($message.message));

  set_field("snort_alert", false);

  set_field("sd_host", m["0"]);
  set_field("sd_proto", m["1"]);
  set_field("sd_src", m["2"]);
  set_field("sd_sport", m["3"]);

  set_field("sd_dst", m["4"]);
  set_field("sd_dport", m["5"]);
  set_field("sd_sid", (m["6"]));
  set_field("sd_rev", m["7"]);

  set_field("sd_desc", m["8"]);
end

Best Answer

The problem might be you need to change the Message Processors Configuration order under Configurations screen.

By default it's GeoIP Processor > Pipeline Processor > Message Filter Chain.

As documented on Greylog official docs about pipelines' usage under "Configure the message processor", you need to change this to Message Filter Chain > Pipeline Processor > GeoIP Processor.

I forgot to do this on a newly installed 2.2 Graylog after having it work for some weeks under 2.1. Symptoms were the same as yours, the snort stream rules and pipeline rules matched, simulations worked, but no fields were actually set and no sign of errors, yet curiously it would work if I connected the pipeline to the default "All messages" stream, which I don't want. Now I've changed this message processor order the snort pipeline works as expected.

Related Topic