monitor or watch a process and restart if it goes down

Question

I have a worker program which I start in rc.local. The program is not the most stable in the world so I would like to set up a watcher of some kind that checks to see if the aforementioned program is chugging along and if not, start up the program again.

I can imagine a couple of ways to do this, although I am sure there are more:

1. have a cron job that runs and looks for the running program. If there, it stops, otherwise it fires off another instance of the program
2. run a dedicated monitoring program that runs all the time and polls frequently to see if the program is running.

Do any of you have examples of either of these implementations or recommendations on how I should do this?

Solution 1

I'd suggest Monit. You can install it from Synaptic or sudo apt-get install monit. I use it on my server to monitor my SSH, Apache, etc and restart them if they fail.

Solution 2

You might be able use a helper script that runs the process in an infinite loop, e.g.:

#!/bin/bash

# Infinite loop
while true; do

    # Get starting time
    start_epoch=$(date +%s)

    # Run program
    my-program

    # Abort if the application exited too quickly
    duration=$(( $(date +%s) - $start_epoch ))
    if [[ "$duration" < 30 ]]; then
        echo "Program exited too quickly. Aborting relauncher."
        exit
    fi

done

exit

Call this script instead of my-program in /etc/rc.local.

For more complex example, see this script which I use to automatically restart the NetworkManager Applet.

Comments

[JD Long]

Today I was notified that this question has 2500 views and I earned a "Notable Question" badge. This seems juxtaposed to the logic behind closing the question. Empirically this question is interesting for future visitors.

[Oki Erie Rinaldi]

I am looking for the questions like this since 2 days ago. I have a process that needs to be working non-stop, but the process is not stable, it could be failed/stopped anytime. Which answer worked for you?