I'm trying out Graylog for system logs and Snort alerts. I've followed the example here to get my snort alerts into Graylog and then proceeded to add another Stream, Pipeline and Rule for a separate IDS log source. I basically copied the Snort example and changed the Regex to extract the pertinent fields from the new log source. The new Regex does test OK when I select a message from the search tab and then select "Test against stream". Messages are also showing up under my new stream when selected from the Streams
menu item at the top of Graylog.
The thing is, in the Pipeline rule, I use the set_field()
function to assign the value of the matching group from the regex. I've got nine fields in the Rule using set_field()
however none of them are showing up in the search data. Why is this? Do I need to use add_field()
first like in the GELF examples? I was assuming set_field()
automatically did that as that is how the snort example at the link above works. Attached my Pipeline rule below which is attached to my IDrops stream in Graylog.
rule "Extract IDropS fields"
when
has_field("message")
then
let m = regex("^([a-z]+)\\s.*(TCP|UDP|ICMP)\\s([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}):(\\d{1,5})\\s([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}):(\\d{1,5}).*SnortSid.*:([0-9]+):([0-9]+)\\s(.*)$", to_string($message.message));
set_field("snort_alert", false);
set_field("sd_host", m["0"]);
set_field("sd_proto", m["1"]);
set_field("sd_src", m["2"]);
set_field("sd_sport", m["3"]);
set_field("sd_dst", m["4"]);
set_field("sd_dport", m["5"]);
set_field("sd_sid", (m["6"]));
set_field("sd_rev", m["7"]);
set_field("sd_desc", m["8"]);
end
Best Answer
The problem might be you need to change the
Message Processors Configuration
order underConfigurations
screen.By default it's
GeoIP Processor > Pipeline Processor > Message Filter Chain
.As documented on Greylog official docs about pipelines' usage under "Configure the message processor", you need to change this to
Message Filter Chain > Pipeline Processor > GeoIP Processor
.I forgot to do this on a newly installed 2.2 Graylog after having it work for some weeks under 2.1. Symptoms were the same as yours, the snort stream rules and pipeline rules matched, simulations worked, but no fields were actually set and no sign of errors, yet curiously it would work if I connected the pipeline to the default "All messages" stream, which I don't want. Now I've changed this message processor order the snort pipeline works as expected.