Novell Cluster services monitoring with Nagios

I recently divided files by business area, then into staging/production, but that required too much prior knowledge from other colleagues to just find something and change it. Also we moved some servers around and I realised it would be useful to have config-per-host files that could be moved as well. Also, having config-per-host allows you to neatly encapsulate all the monitoring for a particular host (or hostgroup) in one file, and see easily what you are doing for that host. On top of that we have a central commands.cfg file, divided up by each business area which can be easily searched.

However none of this is truly satisfactory - using flat-file storage for configuration commits you to favouring one particular ordering method over another. This sort of configuration really benefits from the relational model. Most new monitoring products have implemented configuration from a database for this reason, I guess.

Currently NDOUtils (warning: pdf) gets your config and event data into a database, but as far as I know there's no way to tell Nagios to use that database as config source. It does give you the opportunity to create tables aside from those required by NDOUtils, perhaps creating a separate nagios_config database where you can map things out and then script up a solution that pulls this information out of the database and creates your config.

I'm going to try and use Puppet to generate my Nagios configs in the near future, hopefully it will let me resolve these sorts of issues.

Alright, Nagios 101

Your Nagios installation is most likely in /usr/local/nagios/

Underneath that, you've got a lot of directories, but the one that the plugins go into is libexec. If you 'cd' into there, do an 'ls' and check it out. Those are all the plugins that you can use right now. Most, if you run them by themselves, will give you example usage.

Incidentally, if you know how to program, you can write your own scripts to check whatever you want. The language of the script doesn't matter, as long as the machine can run it. I write mine in bash, lots of people use perl, and the ones that come by default are (mostly?) written in C.

Anyway, once you've got the plugin into the directory, you've got to tell Nagios about it. Go to '/usr/local/nagios/etc/'. If you've got a recent installation (ie 3.0 or above) you should have an 'objects' directory. Inside there, by default, you'll have a "commands.cfg". Edit that, and check out the existing entries. They're all of the format

define command{
    command_name    command_name
    command_line    command_line

}

Put your new plugin in at the bottom.

define command{ 
    command_name      my_mysql_check
    command_line      $USER1$/whatever
} 

The "$USER1$" is a Nagios macro that points to the /usr/local/nagios/libexec directory. You can edit the "resources.cfg" to see what else is available, and even add macros.

Anyway, now that we've got the command in there, we've got to set up a service to take advantage of it.

Save the commands.cfg and edit "services.cfg". Notice that everything is in the format:

define service{
     host_name server
     service_description whatever
     etc 
     etc
} 

These set up the actual service description and the "host_name" assigns it to a server. Look at the documentation here when creating your service. The stuff in red is necessary, the others are used to refine how your service check behaves.

I hope this helped in some way. If not, please reply and let me know. You should also know that the default layout for Nagios configs is simple but unproductive. I wrote some documentation on how I lay mine out, and it's saved me a lot of time trying to find exactly the definition I was looking for. Depending on your installation, it may be overkill.

http://www.standalone-sysadmin.com/blog/2009/07/nagios-config/

Good luck!