Simple way of restarting crashed processes?

Question

I need to monitor several processes running on my webserver. For some reason, varnish currently crashes once every day or two. I'm using monit to supposedly restart varnish automatically, but it doesn't work. Here's my monit.conf entry for Varnish.

check process varnish with pidfile /var/run/varnish.pid
    start program = "/etc/init.d/varnish start" with timeout 60 seconds
    stop program = "/etc/init.d/varnish stop"
    if failed host <my server ip> port 80 protocol http
        and request "/blank.html" then restart
    if 3 restarts within 5 cycles then timeout
    group server

The log file shows that after varnish stops running, the attempted restarts afterwards all fail. Then eventually monit stops monitoring varnish.

Anyone have suggestions for how I can fix this? Or better yet, can you suggest other simple ways of automatically monitoring and restarting crashed processes? Thanks!

Solution 1

I'd look in to daemontools (http://cr.yp.to/daemontools.html).

Supervise was built for exactly this purpose -- to start processes and watch them, restarting them immediately if they ever terminate.

You could still use monit if you need to do anything more complicated than a simple "is it still running" check, and if the process needs to be restarted, then do that through supervise.

Solution 2

You could also use /etc/inittab to restart dead processes using the respawn action.

See inittab section on http://aplawrence.com/Unixart/startup.html

Solution 3

Another great method taken from StackOverflow:

until myserver; do
    echo "Server 'myserver' crashed with exit code $?.  Respawning.." >&2
    sleep 1
done

This could be added to the crontab:

crontab -e

Then add a rule to start your monitor script:

@reboot /usr/local/bin/myservermonitor

Or added as a script in /etc/init.d

See the StackOverflow answer for a detailed explanation on why this is a good approach.

Solution 4

You can use event handler scripts with Nagios if you have that in place to restart services.

If varnish requires root permission to start (init.d scripts usually do) change "/etc/init.d/varnish start" to "sudo /etc/init.d/varnish start". But that probably won't be quite enough since you probably don't want to give whatever user monit runs as total sudo nopasswd privileges to all commands and giving sudo to a shell script would be basically just as bad. So you are going to need to figure out which commands in that init script need sudo, give those commands sudo privileges in the /etc/sudoers file to the monit user, and the finally edit that init script accordingly. Or maybe instead of all this varnish can be run as non-root user?

Finally, I am sure you know this but I am going to say it anyways. You are clearly putting a lot of effort into this, I hope you are putting as much effort into figuring out why varnish is crashing and actually fixing it (or hounding the developers to figure out why) :-)

Update:
This might not be as clean, but an easy way to get this done as root might be to set up a script that checks if the process is okay, and if not starts it. Then just run that script every couple minutes as a cron job.

Solution 5

I was also looking for the most simple way to handle this problem. The easiest way i could find is to simply add Restart=always to the concerning .service file in /etc/systemd/system/multi-user.target.wants/ as last line of the [service] tag.

After that do sudo systemctl daemon-reload followed by sudo systemctl restart service.service to reload the changes.

You can test by checking if the service is running: systemctl status processname, check the start timestamp. After that do ps -ef | grep servicename, ad kill the process with the just found id kill 1234. after that do systemctl status processname again and check if the start timestamp is updated.

Also check /var/log/daemon.log for errors

It should work on:

• Debian 7 and Debian 8
• Ubuntu 15.04 and newer
• CentOS 7 and newer

View more solutions

Comments

[Fl0v0]

i cant believe how difficult such things were in pre-systemd times.

[edomaur]

I use the daemontools too, for monitoring unstable services processes. Quite handy if i had to say. :-)

[brianjob]

I considered Nagios at first, but wanted something small and simple for my purposes. And yes, I'm looking into the Varnish issue. One of my servers has been running it stable for a very long time, so it's definitely got to do with me. :(