How best to monitor and alert on the absence of an event in the logs

alertsmonitoringscriptingsyslog-ng

With logs getting captured in syslog-ng, I'd like to be able to automatically monitor the logs and receive an alert if NO log events appear that match a certain criteria. For instance, for a subscription-based website, if 6 hours elapse with no orders, then email or text this person or group of people.

What is a good way to do that?

Best Answer

http://labs.consol.de/nagios/check_logfiles is a Nagios plugin which is used to monitor logfiles. Usually you check, if there is a certain pattern (error message). But it's alos possible to reverse this. For example, if you run check_logfiles --logfile /var/log/mybackup.log --criticalpattern '!backup succeeded' every morning, you will get an alert if there was no 'backup succeeded' message entry since the last run of check_logfiles.

Gerhard

Trap Form (OID)

These traps will conform to the Microsoft private enterprise MIB branch in the following form:

1.3.6.1.4.1.311.1.13.X.n.n.n.n.n.n.n.n.n...

Each "n" is a decimal encoding of an ASCII character octet from the Event Log source name, and the X designates the number of characters to follow.

So, for example, a trap generated by source "Prefect" (as seen in Event Viewer) would appear as:

1.3.6.1.4.1.311.1.13.7.80.114.101.102.101.99.116

Windows 2000 Server does not support this fully, and will generate traps of a slightly different format, but the procedure is otherwise identical. All newer versions of Windows server support this properly

Configuring Trap Sending

There are two built-in tools that you will use to set up trap generation.

evntwin: Create mapping of Event Log messages to SNMP traps evntcmd: Load mapping created by evntwin so that traps are generated

Run evntwin from a command prompt: this will spawn a GUI. Select "Custom" under Configuration type, and then "Edit." You will now see a list of all possible event sources. Under the source in which you are interested, select the particular event ID on which you wish to generate traps. Then, click "Add."

Now, you will see the actual OID of the trap, the specific ID, and an option to set a time-based threshold of event occurrences before the trap would be sent.

Repeat until you have created a mapping for each particular trap/event combination you care about. Then, click "Apply," highlight all of the mappings, and then "Export..." Save the file, and exit the application.

Now, again from the command line, run evntcmd, specifying the name of the file you just created:

evntcmd myeventfile.cnf

From this point forward, the events you specified will generate SNMP traps, which will be sent to all trap receiver destinations you have configured in your SNMP service settings. Process them as you would any normal SNMP trap.

Linux monitor logs and email alerts

You could use something like LogWatch. Or even a simple script like this (it's pseudo code you'll need to modify it for your enviroment):

 #!/bin/bash
 GREP_STRING=`grep -c <error string> <acpid log location>`
 if [ $GREP_STRING -ne 0 ] 
 then
    <send email notification>
 fi

Put that in cron to run every hour or so and you should get an email letting you know when it's getting wierd.

Best Answer

Related Solutions

Windows – How to Passively Monitor the Windows Event Log

Trap Form (OID)

Configuring Trap Sending

Linux monitor logs and email alerts

Related Topic