Cron – Best way to verify that a cronjob executed with Nagios notification on failure

cronnagios

I have a cron job that runs every morning that produces an output file when run successfully. I'd like to have a follow up process that verifies that the job executed properly or fire off a Nagios alert if it didn't.

I feel like I could make this work with a Nagios check_file_age service check that restricts the check to occur only once every morning through careful construction of the check_interval and check_period in the service def, but I'm wondering if there is a simpler way?

Best Answer

A couple of options spring to mind:

Use passive check results to have the verification script tell Nagios to go alert, and if you want to know if the verification script doesn't run, then set a freshness interval of a little over 24 hours and have an active check command that always goes critical; or
Have the cron job itself submit an OK passive check result when it finishes successfully, but not do anything when it fails. Then, again, freshness check with an auto-critical active check command to alert when all is not well.

Related Solutions

What’s the best way to pair Nagios monitoring with EC2

You can use Dynamic DNS. Install a Dynamic-DNS client on every one of your servers and the IP will be updated when it changes.

You'll have to configure Nagios with hostnames instead of IPs. Probably you want a low TTL on the DNS server to avoid the old IP being cached too long.

Nagios – How to Dynamically Set a New Test Interval for Nagios Checks

You can do it by using CHANGE_NORMAL_SVC_CHECK_INTERVAL and CHANGE_NORMAL_HOST_CHECK_INTERVAL.

Add an event handler for your service:

define service {
    host_name              ...
    service_description    ...
    check_command          ...
    contact_groups         ...
    event_handler          change_check_interval
}

The change_check_interval was defined in commands.cfg:

define command {
    command_name    change_check_interval
    command_line    $USER1$/eventhandlers/change_check_interval.sh $SERVICESTATE$ $SERVICESTATETYPE$ $SERVICEATTEMPT$ $HOSTADDRESS$
}

The content of change_check_interval.sh:

#!/bin/bash

now=`date +%s`
commandfile='/usr/local/nagios/var/rw/nagios.cmd'

case "$1" in
    OK)
        ;;
    WARNING)
        ;;
    UNKNOWN)
        ;;
    CRITICAL)
        /bin/printf "[%lu] CHANGE_NORMAL_SVC_CHECK_INTERVAL;host1;service1;2\n" $now > $commandfile
        ;;
esac

exit 0

Make sure that external commands is enabled in nagios.cfg:

check_external_commands=1

Best Answer

Related Solutions

What’s the best way to pair Nagios monitoring with EC2

Nagios – How to Dynamically Set a New Test Interval for Nagios Checks

Related Topic