Logstash Grok Parsing Failed

groklogstashregular expressions

I need to match this entry

2015/10/30 23:58:21 pid 22223 testuser@testserver.example.com 192.168.0.1 [p4/2012.2/LINUX26X86_64/536738] 'test-monitor show'

To match this I wrote this Regex

P4_DATE (?>\d\d){1,2}\/(?:0[1-9]|1[0-2])\/(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
P4_TIME (?:2[0123]|[01]?[0-9])\:(?:[0-5][0-9])\:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)
P4_PID \b(?:[1-9][0-9]*)\b
P4_USER \b\w+\b
P4_HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)
P4_IP (?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9])

and then overall

 P4_MATCH %(P4_DATE:p4date} %{P4_TIME:p4time} pid %{P4_PID:p4pid} %{P4_USER:p4user}\@%{P4_HOSTNAME:p4client} %{P4_IP:p4remoteclient} [%{DATA:p4version}] \'%{DATA:p4action}\'" }

and then match via

match => [ "message", "%{P4_MATCH}" ]

But still getting "_beats_input_codec_plain_applied, _grokparsefailure"

I am not a regular expression expert but any help is really appreciated.

Best Answer

I don't see the specific issue, but take a look at https://grokdebug.herokuapp.com/ -- build up your patterns one field at a time and see what step causes matching to fail.

Related Solutions

Zabbix 2.0: Regular Expression to extract value from a String for a trigger

Function regexp returns 1 if the item's value matches the regular expression provided as the argument, and returns 0 otherwise. It does not return the extracted value.

In general, what you are trying to do is currently not possible, but there are two feature requests that may make it possible in the future: ZBXNEXT-1638 (store the whole value, but extract a piece of information later) and ZBXNEXT-1427 (extract a piece of information upon receival).

As a workaround, if the threshold value for ms_time would be a round value (say, 10000), it would be possible to trigger based on ms_time length:

192\.168\.105\.20,[\d]+,([\d]{5,})

Logstash Grok Pattern

Here is a simpler version of a pattern that might help you to get started:

(?<SVN>[0-9]+)\|%{USERNAME:username}\|(?<SVN_TIMESTAMP>[^\|]+)\|%{GREEDYDATA:syslog_message}

For simplicity's sake I did not use named patterns and the timestamp is not very specific at all, but that should be easier to fix.

Important things to note:

the pipe character is a logical OR in these expressions, it needs to be escaped
as @tigran pointed out: you need the plus symbol for "one or more" digits on the SVN revision
your SVN_TIMESTAMP pattern is very complex, but doesn't seem quite right. At a minimum you need to escape the parentheses to match.

I recommended you take your input and my pattern and put it into https://grokdebug.herokuapp.com/ -- that will allow you to gradually enhance it to what you really need.

Related Topic