Dynamically adding new host in nagios

I recently divided files by business area, then into staging/production, but that required too much prior knowledge from other colleagues to just find something and change it. Also we moved some servers around and I realised it would be useful to have config-per-host files that could be moved as well. Also, having config-per-host allows you to neatly encapsulate all the monitoring for a particular host (or hostgroup) in one file, and see easily what you are doing for that host. On top of that we have a central commands.cfg file, divided up by each business area which can be easily searched.

However none of this is truly satisfactory - using flat-file storage for configuration commits you to favouring one particular ordering method over another. This sort of configuration really benefits from the relational model. Most new monitoring products have implemented configuration from a database for this reason, I guess.

Currently NDOUtils (warning: pdf) gets your config and event data into a database, but as far as I know there's no way to tell Nagios to use that database as config source. It does give you the opportunity to create tables aside from those required by NDOUtils, perhaps creating a separate nagios_config database where you can map things out and then script up a solution that pulls this information out of the database and creates your config.

I'm going to try and use Puppet to generate my Nagios configs in the near future, hopefully it will let me resolve these sorts of issues.

What you want in this case is not dependencies (as Nagios calls them), but rather you want parent/child relationships in your hosts. If the parent is down and children can not be reached Nagios considers them "unreachable" instead of "down". You can then configure not to be notified if things are "unreachable". You can read more about this in the Nagios documentation here.

Also, from the same section of the 2.0 documentation it is pointed out that services won't alert if the host is down (saving you those alerts):

Whenever a service check results in a non-OK status level, Nagios will attempt to check and see if the host that the service is running on is "alive". Typically this is done by pinging the host and seeing if any response is received. If the host check command returns a non-OK state, Nagios assumes that there is a problem with the host. In this situation Nagios will "silence" all potential alerts for services running on the host and just notify the appropriate contacts that the host is down or unreachable. If the host check command returns an OK state, Nagios will recognize that the host is alive and will send out an alert for the service that is misbehaving.

Service dependencies by default are not inheritable from a host template, but I believe the parent is. So you can inherit parents from templates (if I am correct in this) and save yourself some typing.

Lastly, I actually like to have unreachable alerts enabled. If the router goes down, I want my phone to go "nuts". :-)