Postgresql – Monitoring PostgreSQL replication with Nagios and check_postgres shows intermittent delay

I did this with a cron script, stores current value in a temp file then next time uses it to calculate bandwidth utilization since last run.

#!/bin/bash

email_address=""
router_ip=""

# 80% BANDWIDTH [ (384000bps) 48,000Bps ] - 20% = 38,400 Bps
alertBW="76800"

lastBWFile="/var/log/ciscoGW.log"
lastBW=`cat $lastBWFile | awk '{print$2}'`
lastTime=`cat $lastBWFile | awk '{print$1}'`

curBW=` snmpget  -c snmap_name -v 1 $router_ip IF-MIB::ifOutOctets.2  | awk '{print$4}'`

let diffBW=$curBW-$lastBW
#echo "Diff BW: $diffBW"
timeNow=`date +%s`
let diffTime=$timeNow-$lastTime
let alertBW=$alertBW*$diffTime

echo "$timeNow $curBW" > $lastBWFile

if [ $diffBW -gt $alertBW ]; then
#       echo "Over limit!"
        echo "Bandwith used over $diffTime seconds: $diffBW" | mail -s "BANDWIDTH OVER LIMIT!!!!" $email_address
fi

Since I was more interested in actual peaks I've since moved to using rrdtool:

#start 15 minutes ago
#end 5 minutes ago since rrdtool queries every 5 minutes 
rrdtool fetch $FROM MAX -s -900 -e -300 

Assuming that the interval_length directive is set to 60 by the default:

$ grep interval_length /usr/local/nagios/etc/nagios.cfg 
# This value works of the interval_length you specify later.  If you leave
# actual seconds rather than a multiple of the interval_length variable.
interval_length=60

For the special services, you need to define a different template for it in /usr/local/nagios/etc/objects/templates.cfg:

define service{
        name                            special-service    
        ...
        max_check_attempts              3           
        normal_check_interval           20         
        retry_check_interval            3           
        notification_interval           60   
        ...   
        }

Pay attention to the:

• normal_check_interval: this service is check every 20 minutes under normal condition
• retry_check_interval: the number of minutes to wait before scheduling a re-check when service has changed to non-OK state. Notice that if the service has been retried max_attempts time without a change in its status, it will revert to being scheduled at check_interval rate.

and use this template for your service:

define service{
    use                     special-service
    host_name               xx
    service_description     yy
    check_command           zz
    contact_groups          admins
    }

You may also need to define a service escalation to change the notification_interval based on the service state, something like this:

define serviceescalation{
    host_name               xx
    service_description     yy
    last_notification       0
    notification_interval   10
    escalation_options      [w,u,c]
    contact_groups          admins
    }

It means that this service escalation is used when service is in WARNING, UNKNOWN, or CRITICAL state. And you now have a new notification intervals: 10 minutes.