Monit is not restarting process

Question

I have monit to monit a process but it doesn't realize it is down. My monit configuration is this:

 check process xxx with pidfile /var/run/xxx.pid
 group yyy
 start program = "/etc/init.d/xxx start"
 stop program = "/etc/init.d/xxx stop"
 if failed host 127.0.0.1 port 9999  then restart
 if 5 restarts within 5 cycles then timeout

The problem is that the PID file still exists even if my process has crashed. So...PID exists but port 9999 in localhost should fail. So...is it expected for monit not to restart this process? Should be delete the pid file for sure? Cannot make that kind of an OR?

Thanks in advance,

Solution

Correct me if i am wrong :

• if program crashes, PID file stays but port 9999 is closed
• if program stops cleanly, PID file is removed and port 9999 is closed

The common behaviour is about port 9999. You could just monitor this using something like this :

check host 127.0.0.1 with address 127.0.0.1
  start program = "/etc/init.d/xxx restart"
  stop program = "/etc/init.d/xxx stop"
  if failed port 9999 then restart

The above should be sufficient to cover all cases, but you can add it as an additional statement to your existing one (that will be the kind of OR you are talking about).

Have a look at the doc regarding check host.

PS : I think, regarding the doc (the sample about Apache), that the config you have should be able to restart your process if port 9999 is not reachable, even if PID file still exists. My guess is that port 9999 is not closed, or port 9999 is not listening on 127.0.0.1 (please check all these using netstat).

So, in your config, try to replace the port monitor line by this one :

if failed port 9999 then restart (removing host 127.0.0.1).

Comments

[Mariano Martinez Peck]

Thanks, the first configuration you pasted works as expected. It is strange still why Monit would not restart with my script since I am sure port was not open...

[krisFR]

@Mariano Martinez Peck Another thing that i've noticed is that sometimes, using /etc/init.d/xxx start for the start program does not work, but when using restart instead of start works...Another thing i've forget to ask is : did you check Monit log file to see if the restart event was triggered when process crashes ?

[Mariano Martinez Peck]

As far as I remember I didn't even see the even in the log. Also, I have email notification when the process fail, and I didn't receive any email either in those cases.