Distribute Nagios to reduce false alarms

Ok I finally got it working.

The problem was that both nagios servers are performing service checks and reporting results to master node, and all those check were performed perfectly. Master node had service freshness checking so if the monitoring servers could not complete checks master server would scheduled those checks from itself.

Anyways, new servers were on new ip range and by default nrpe port was closed on master server.

Opening the port solved the problem. Although it's still odd that it returned "Service check timed out" instead of "Socket timeout error".

You can define a custom object variable, something like this:

define host {
        use                     generic-host
        host_name               server1
        alias                   server1.gertvandijk.net
        address                 10.1.2.3
        _wan_address            1.2.3.4
        }

then call it as $_HOSTWAN_ADDRESS$ macro:

define command{
        command_name    check_wan_http
        command_line    $USER1$/check_http -I $_HOSTWAN_ADDRESS$ $ARG1$
        }

define service{
    use                     generic-service
    hostgroup_name          apache
    service_description     Apache
    check_command           check_wan_http
    process_perf_data       0
    contact_groups          admin
}