Nagios Woudn't Start, now won't Stop!

Question

I ran an update on a CentOS server running Nagios, after the update, Nagios failed to start.

The error in the logs was:

Failed to obtain lock on file /var/run/nagios.pid: Permission denied

So, I checked and there was no pid file for Nagios in /var/run. I created one and gave it the following permissions:

-rwxr--r-- 1 nagios nagios 6 May 31 11:58 nagios.pid

Nagios then started and seems to be running normally.

The only problem is, it refuses to stop now, so I can't re-start it to add new servers and services to be monitored!

When I issue the command "service nagios stop", I get [FAILED], but nothing at all gets outputted to the log, and the service remains up.

Any ideas on how I can get the service to stop now?

I'm running the RPM version which was installed via yum from the RPMForge repositories. The server is CenotOS 5.5.

Solution 1

If you don't want to change the permissions on the /var/run folder itself, you can configure nagios to store the lock file anywhere on the system by adding the line below to your nagios.cfg file. As long as you point the lock_file option to a directory that nagios has access to create, modify and delete files then you should be good to go.

You could even store the lock file within your nagios installation directory by creating a a directory like: /usr/lib/nagios/var

lock_file=/usr/lib/nagios/var

Solution 2

First, find out what the pid of nagios process is:

$ ps aux | grep nagios

Then you can use this command to restart the service:

kill -HUP <nagios_pid>

and this to stop:

kill <nagios_pid>

To stop nagios automatically , you should have its pid in /var/run/nagios.pid. Check it.

Solution 3

As Bart B said above, do not chmod 777 any directory on a production system, ever, unless there is a really good reason for it and you know what you are doing. This was a quick fix, but this is not the solution.

The correct solution to this problem, at least for me, was to update the nagios.cfg Nagios configuration file (mine is /etc/nagios/nagios.cfg), and to change this line:

   lock_file=/var/run/nagios.pid

to this:

   lock_file=/var/nagios/nagios.pid

The lock file is set to /var/nagios/nagios.pid in the /etc/init.d/nagios service configuration file, but is apparently overridden by the above value in nagios.cfg

Then you may restart the Nagios service/daemon:

   service nagios restart

That should do it.

Comments

[Govindarajulu]

What is the contents of the pid file?

[Walter K]

The PID file contained the correct PID.

[lexsys]

I don't think it's a good idea.

[Govindarajulu]

Do not chmod 777 anything, ever, unless you have a really good reason to do so. Which this is not.

[Walter K]

Yikes - I guess it will work in the sense that Nagios will start and stop, but man, too dangerous for my blood! I certainly wouldn't recommend this on a production system!

[Walter K]

Excellent - with just a small change this is exactly what I needed. When I went to edit nagios.cfg I found a nagios.cfg.rpmnew file there too, and in there was a different location for the pid file: lock_file=/var/nagios/nagios.pid Once I coppied that over to nagios.cfg all was well

[PiL]

I know it's not a good idea, but it's a way to understand the problem. Because sure something in centos changed something and infact if you restart the machine the /var/run comes back to the original permission and nagios doesn't work again. As John Rabotnik suggested you can change where the lock file is written (and is what i did myself too) but when nagios will be updated the problem will come back again (or you keep a copy of the init file and use always the old one). If you login with the nagios user you can write normally in /var/run with the original permission.

[PiL]

So it looks like when you run an init script something is blocking nagios to write inside that directory. It's not easy to understand why it doesn't work. That's why i suggested a simple and fast workaround to change permission on the dir.

[hurfdurf]

In my case, the /etc/init.d/nagios pidvalue was not overridden by nagios.cfg, which caused "service nagios stop" to fail. Setting both nagios.cfg and /etc/init.d/nagios pidfile values to the same filepath solved the problem.

[lynxman]

He says he tried already, that's not the answer ;)

[lynxman]

One of those things to never ever do, not even for a suggestion :/

[Frank Zhang]

For Stop: (1) ps aux | grep Nagios ( 2)kill <process ID #>

[Ale]

Well, I'd say you can do it, but only in a testing environment, never on a production machine.