How to automatically restart a linux background process if it fails?

Question

I have a process which is executed by init.d script on the background. Eg:

case "$1" in 
    start)
       /bin/myprocess &
    stop)
       killall myprocess
    restart)
       killall myprocess
       /bin/myprocess &
esac

In certain conditions, myprocess can fail and return. Is there any (standard) way how to detect its fail and restart in automatically?

Solution 1

The easiest way would be to add it to /etc/inittab, which is designed to do this sort of thing:

respawn If the process does not exist, start the process. Do not wait for its termination (continue scanning the /etc/inittab file). Restart the process when it dies. If the process exists, do nothing and continue scanning the /etc/inittab file.

For example, you could do this:

# Run my stuff
myprocess:2345:respawn:/bin/myprocess

Solution 2

Buildroot has three possible init systems, so there are three ways to do this:

BusyBox init

With this, one adds an entry to /etc/inittab.

::respawn:/bin/myprocess

Note that BusyBox init has an idiosyncratic /etc/inittab format. The second field is meaningless, and the first field is not an ID but a device basename.

Linux "System V" init

Again, one adds an entry to /etc/inittab.

myprocess:2345:respawn:/bin/myprocess

systemd

One writes a unit file in, say, /etc/systemd/system/myprocess.service:

[Unit]
Description=My Process

[Service]
ExecStart=/bin/myprocess
Restart=always

[Install]
WantedBy=multi-user.target

Enable this to autostart at bootup with:

systemctl enable myprocess.service

Start it manually with:

systemctl start myprocess.service

Further reading

• "3.1.3 init system". The Buildroot user manual.

Solution 3

What about creating a subshell with a loop that calls constantly the same process?

If it ends, the next iteration of the loop goes on and starts it again.

(while true; do 
    /bin/myprocess
done) &

If the subshell dies, it's over though. The only possibility in that case would be to create another process (I'll call it necromancer) that checks whether yourprocess is alive, start it if it isn't and run this necromancer with cron, so that you can check that regularly.

Next step would be wondering what could happen if cron dies, but at some point you should feel safe and stop worrying.

Solution 4

You could make use of Monit . It's really easy to use and quite flexible. See for example this configuration for restarting the Tomcat process on failure.

check process tomcat with pidfile /var/run/tomcat.pid
   start program = "/etc/init.d/tomcat start"
   stop  program = "/etc/init.d/tomcat stop"
   if failed port 8080 type tcp then restart

It also has a lot of configuration examples for many use cases.

Solution 5

You may use restarter

start)
   restarter -c /bin/myprocess &
stop)
   pkill -f myprocess

On newer systems use systemd which solves all those trivial issues

View more solutions

Comments

[David Schwartz]

Sure, but it varies based on distribution. Pretty much all of them provide some kind of service manager.

[Honza]

There is no standard distribution, but buildroot. So I have to do it manually...

[peterh]

Note, /etc/inittab works (or even exists) if and only if you have a sysvinit-based init system. With upstart and with systemd it doesn't. You have to install either busybox (very primitive shell making the sysadm recover tasks painful, but it can substitute a sysvinit-compatible initd) or sysvinit (it is a fossil). In a docker container, only the first is not painful.

[horseyguy]

but when you use this approach inittab then your process is no longer accessible via the 'service' interface right? i.e you can't go service mything start or service mything stop anymore....is there a way to have the best of both? i.e uncrashable sysvinit service, but also have it usable via 'service' ?

[kenneth558]

Running systemd, so the obvious choice I would have is #3 above. Yet for some unknown reason, my nohup ffmpeg ... & daemon won't save its files when launched this way, so I have to start it manually for it to work. Out of 4 similar daemons, rarely do all four continue 24 hours. Usually one fails for unknown reason[s] and sometime two, meaning I really need this auto-restart. I'm upvoting one of the options below until I should find it doesn't work.

[Dolf Andringa]

He is asking about init.d scripts. Adding docker on top is really not the tool to auto restart a daemon. It's like automating a computer power cycle if the daemon stops.