Python script as linux service/daemon

Question

Hallo,

I'm trying to let a python script run as service (daemon) on (ubuntu) linux.

On the web there exist several solutions like:

http://pypi.python.org/pypi/python-daemon/

A well-behaved Unix daemon process is tricky to get right, but the required steps are much the same for every daemon program. A DaemonContext instance holds the behaviour and configured process environment for the program; use the instance as a context manager to enter a daemon state.

http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/

However as I want to integrate my python script specifically with ubuntu linux my solution is a combination with an init.d script

#!/bin/bash

WORK_DIR="/var/lib/foo"
DAEMON="/usr/bin/python"
ARGS="/opt/foo/linux_service.py"
PIDFILE="/var/run/foo.pid"
USER="foo"

case "$1" in
  start)
    echo "Starting server"
    mkdir -p "$WORK_DIR"
    /sbin/start-stop-daemon --start --pidfile $PIDFILE \
        --user $USER --group $USER \
        -b --make-pidfile \
        --chuid $USER \
        --exec $DAEMON $ARGS
    ;;
  stop)
    echo "Stopping server"
    /sbin/start-stop-daemon --stop --pidfile $PIDFILE --verbose
    ;;
  *)
    echo "Usage: /etc/init.d/$USER {start|stop}"
    exit 1
    ;;
esac

exit 0

and in python:

import signal
import time
import multiprocessing

stop_event = multiprocessing.Event()

def stop(signum, frame):
    stop_event.set()

signal.signal(signal.SIGTERM, stop)

if __name__ == '__main__':
    while not stop_event.is_set():
        time.sleep(3)

My question now is if this approach is correct. Do I have to handle any additional signals? Will it be a "well-behaved Unix daemon process"?

Solution 1

Assuming your daemon has some way of continually running (some event loop, twisted, whatever), you can try to use upstart.

Here's an example upstart config for a hypothetical Python service:

description "My service"
author  "Some Dude <[email protected]>"

start on runlevel [234]
stop on runlevel [0156]

chdir /some/dir
exec /some/dir/script.py
respawn

If you save this as script.conf to /etc/init you simple do a one-time

$ sudo initctl reload-configuration
$ sudo start script

You can stop it with stop script. What the above upstart conf says is to start this service on reboots and also restart it if it dies.

As for signal handling - your process should naturally respond to SIGTERM. By default this should be handled unless you've specifically installed your own signal handler.

Solution 2

Rloton's answer is good. Here is a light refinement, just because I spent a ton of time debugging. And I need to do a new answer so I can format properly.

A couple other points that took me forever to debug:

1. When it fails, first check /var/log/upstart/.log
2. If your script implements a daemon with python-daemon, you do NOT use the 'expect daemon' stanza. Having no 'expect' works. I don't know why. (If anyone knows why - please post!)
3. Also, keep checking "initctl status script" to make sure you are up (start/running). (and do a reload when you update your conf file)

Here is my version:

description "My service"
author  "Some Dude <[email protected]>"

env PYTHON_HOME=/<pathtovirtualenv>
env PATH=$PYTHON_HOME:$PATH

start on runlevel [2345]
stop on runlevel [016]

chdir <directory>

# NO expect stanza if your script uses python-daemon
exec $PYTHON_HOME/bin/python script.py

# Only turn on respawn after you've debugged getting it to start and stop properly
respawn

Comments

[tauran]

You right, upstart is the standard nowadays! As the above script handles SIGTERM it should be ok with your config file :)

[Anthony Briggs]

One extra tweak that I made just now. If your python script runs under a virtualenv, you just need to change upstart to use the python executable from the environment: exec /home/user/.env/environ/bin/python /some/dir/script.py

[Scott Willeke]

Great info. Where is the documentation for the /etc/init files?

[tiktak]

From upstart version 1.4, you can use "setid" and "setgid". The argument is the user / group name.

[David]

When i copy this config into /etc/init then type service myservicename it doesn't find it. Whats up with that.

[Ross R]

Did you do initctl reload-configuration followed by service myservice start?

[David]

Sorry about that, it works now. I was typing service servicename thinking it wouod give me my options apparently not....you have to do servicename start stop or restart. Thanks for the reply anyway

[yhager]

Thank you much. That tip about 'no expect stanza' is golden. It's unfortunate that upstart keeps some state between restarts and would sometimes hang with the right file for no reason. Makes debugging very hard.

[Hibuki]

Upstart doesn't seem to be the standard anymore. Wikipedia lists many "Linux distributions that [...] moved away since or no longer use it [upstart] as their default init system". They are using systemd instead. More on Upstream vs Systemd from unix.stackexchange