Nagios Won’t Start or Stop – Troubleshooting Guide

centoslinuxnagios

I ran an update on a CentOS server running Nagios, after the update, Nagios failed to start.

The error in the logs was:

Failed to obtain lock on file /var/run/nagios.pid: Permission denied

So, I checked and there was no pid file for Nagios in /var/run. I created one and gave it the following permissions:

-rwxr–r– 1 nagios nagios 6 May 31 11:58 nagios.pid

Nagios then started and seems to be running normally.

The only problem is, it refuses to stop now, so I can't re-start it to add new servers and services to be monitored!

When I issue the command "service nagios stop", I get [FAILED], but nothing at all gets outputted to the log, and the service remains up.

Any ideas on how I can get the service to stop now?

I'm running the RPM version which was installed via yum from the RPMForge repositories. The server is CenotOS 5.5.

Best Answer

If you don't want to change the permissions on the /var/run folder itself, you can configure nagios to store the lock file anywhere on the system by adding the line below to your nagios.cfg file. As long as you point the lock_file option to a directory that nagios has access to create, modify and delete files then you should be good to go.

You could even store the lock file within your nagios installation directory by creating a a directory like: /usr/lib/nagios/var

lock_file=/usr/lib/nagios/var

Related Solutions

Linux – xinetd won’t start

Use "strace" command to debug xinetd, example:

yum install strace
strace /usr/sbin/xinetd 2>&1 | tee log.txt

Then examine log.txt to see what's going wrong with your configuration.

I would recommend you to backup your configurations, uninstall xinedt, remove /etc/xinetd.conf and /etc/xinetd.d/ and then install it again using yum. When you'll be able to start it you can add more services step by step.

The correct way to stop the nagios ndo2db daemon

I found the culprit - it was Centreon's config file builder.

ndo2db has a lock_file setting which is missing from the Centreon config UI.

When Centreon generates the config files it also generates ndo2db.cfg - but without the lock_file configuration value.

There's an open issue about this:

No lock_file field in Configuration - Centreon - ndo2db.cfg

Having spelunked the source code, when ndo2db daemonises and if there isn't a lock_file setting then it ignores this and carries on and no lock file is written containing the PID.

This of course means that the stop function in the init script won't be able to identify the ndo2db process id so it can be killed.

Update:

To resolve this issue I manually added a new column to the cfg_ndo2db table in the centreon database:

ALTER TABLE `cfg_ndo2db`
    ADD COLUMN `lock_file` VARCHAR(255) NULL DEFAULT NULL;

I then populated it with the path of my ndo2db lock file:

UPDATE `cfg_ndo2db` SET `lock_file`='/usr/local/nagios/var/ndo2db.lock' WHERE `id`=1;

This will force centreon to write the lock_file setting each time the config is generated. This also appears to survive upgrades as well, though I'd always check the database upgrade scripts to ensure this doesn't sneak in as an undocumented fix.

Best Answer

Related Solutions

Linux – xinetd won’t start

The correct way to stop the nagios ndo2db daemon

Related Topic