Monit: use content MATCH on “check file …” if match “xxx”

monitregex

I've asked this question around a week a go at StackOverflow but no answer just yet, probably is no possible but don't know where to look for that answer, I hope someone can help over here.

I'm using monit to scan logs for errors and then push those alert into a monitoring system called DataDog…

All seem to work as expected but now I've need to grab what is causing the alarm.

Using a very simple rule I'm able to catch line on log that provoke this error and run the specific script to alert; until here all ok:

montirc file:

check file testmonit with path /var/log/testmonit.log
      if MATCH "(ERROR.*)" then
         exec "/usr/bin/python /opt/scripts/bin/dd_notify.py test-error"

This config make what I want, it actually raise the alarm I wanted

But now I need to know "What caused this alarm"; so for example if this line appears on the log:

ERROR failure to complete process due lock file....

On monit logs I can see:

[UTC Mar  6 11:59:08] error    : 'testmonit' content match [ERROR failure to complete process due lock file....]
[UTC Mar  6 11:59:08] info     : 'testmonit' exec: /usr/bin/python

Which is perfect… what I want is capture this:

[ERROR failure to complete process due lock file....]

To send this string into my monitoring system (DataDog) I can't find any documentation that actually allows me to use the MATCH content, or groups (which I can see is being supported by the MATCH regex)

So in short:

There's any monit variable(like $DESCRIPTION for mail) that refer to MATCH line that trigger the rule?

(I've tried $DESCRIPTION, $HOST…etc but this seems to work only for email)

I'd a look into google several times (and also here) but I can't find the answer…

Please if you think this has been addressed before feel free to point me to right direction.

An update:

Sorry I forgot to say that I'm running this on:

 Ubuntu 16.04 LTS and 
 Ubuntu 12

Monit version is:

This is Monit version 5.25.1
Built with ssl, with ipv6, with compression, with pam and with large files
Copyright (C) 2001-2017 Tildeslash Ltd. All Rights Reserved.

Following DevOps reccommendation, I've upgraded monit version.

And tried to use MONIT_DESCRIPTION or $MONIT_DESCRIPTION without success rule file is like this:

check file pd-error with path /var/log/testmonit.log
if CONTENT = "ERROR" then exec "/usr/bin/python /opt/scripts/bin/dd_notify.py pd_error " $MONIT_DESCRIPTION

What I want is pass the content that has been matched as an additional argument for dd_notify.py program;

but what i get is (which is the result of executing dd_notify.py):

{
"ALARM": {
"pd_error": 67
},
"MESSAGES": {
"pd_error": "$MONIT_DESCRIPTION"
}
}

what I want is $MONIT_DESCRIPTION content which is actually:

[UTC Apr 3 21:53:22] debug : 'pd-error' Pattern 'ERROR' match on content line [Apr 3 21:52:30 ams01 MainProcess[1376]: cel
ery.worker.job ERROR Task tasks.telemetry.gather_and_send_telemetry_info[f090d579-9ec2-40e5-9fb2-91436eb4fc8a] fail]

But I'm not having any luck at the moment… what I'm missing here?

Thank you.

Best Answer

Jsut tried with Monit 5.23.0 and there is an environment variable for it.

MONIT_DESCRIPTION=content match:
[ERROR failure to complete process due lock file....]
[ERROR failure to complete process due lock file....]

Monit will output every occurrence of the matching content

Also the syntax as changed in Monit 5.16.0 but old is still functional. Changelog is available here : https://mmonit.com/monit/changes/

Also be aware that you can change the various limits of Monit https://mmonit.com/monit/documentation/monit.html#LIMITS

If you are willing to upgrade, Monit provides precompiled generic binaries that I'm using to have a newer versions than the ones in the Ubuntu repositories.

Related Topic