Nagios check for service not if host is up

Ok I finally got it working.

The problem was that both nagios servers are performing service checks and reporting results to master node, and all those check were performed perfectly. Master node had service freshness checking so if the monitoring servers could not complete checks master server would scheduled those checks from itself.

Anyways, new servers were on new ip range and by default nrpe port was closed on master server.

Opening the port solved the problem. Although it's still odd that it returned "Service check timed out" instead of "Socket timeout error".

Assuming that the interval_length directive is set to 60 by the default:

$ grep interval_length /usr/local/nagios/etc/nagios.cfg 
# This value works of the interval_length you specify later.  If you leave
# actual seconds rather than a multiple of the interval_length variable.
interval_length=60

For the special services, you need to define a different template for it in /usr/local/nagios/etc/objects/templates.cfg:

define service{
        name                            special-service    
        ...
        max_check_attempts              3           
        normal_check_interval           20         
        retry_check_interval            3           
        notification_interval           60   
        ...   
        }

Pay attention to the:

• normal_check_interval: this service is check every 20 minutes under normal condition
• retry_check_interval: the number of minutes to wait before scheduling a re-check when service has changed to non-OK state. Notice that if the service has been retried max_attempts time without a change in its status, it will revert to being scheduled at check_interval rate.

and use this template for your service:

define service{
    use                     special-service
    host_name               xx
    service_description     yy
    check_command           zz
    contact_groups          admins
    }

You may also need to define a service escalation to change the notification_interval based on the service state, something like this:

define serviceescalation{
    host_name               xx
    service_description     yy
    last_notification       0
    notification_interval   10
    escalation_options      [w,u,c]
    contact_groups          admins
    }

It means that this service escalation is used when service is in WARNING, UNKNOWN, or CRITICAL state. And you now have a new notification intervals: 10 minutes.