How do I set up a daemon with python-daemon?

Question

I'm new to daemons so apologies if this is a newbie question.

In several other answers (for example, this question) people suggested the python-daemon package was the way to go because it fully implements the PEP 3143 standard.

Unfortunately, python-daemon is a bit light on documentation (or more likely I am a bit light on knowledge / experience... ;) ), and I think I am probably missing something really basic. Here's what I'm doing:

I have the following:

import daemon

logfile = open('daemon.log', 'w')

context = daemon.DaemonContext(stdout = logfile, stderr = logfile)

context.open()

with context:
    do_something_1()
    do_something_2()

Question: How do I set up a daemon with python-daemon, how can I start it and stop it?

---

Side notes:

I'm basically taking a wild guess about how / whether the .open() method should be used here -- docs were not real clear on this point. Same thing seems to happen whether I include it or not.

So, now what do I do? When I try running this file, eg:

python startConsumerDaemons.py

it appears to run do_something_1(), but not the second. And, it appears to leave the program attached to the terminal window. IE, stdout isn't redirected, and when I close the terminal window the process is killed. So, I'm pretty sure I'm doing something wrong here... what should I be doing differently?

And, lastly, once I get the daemon running, how do I stop / restart it (for example if I make changes to the underlying code)?

Solution 1

Here is what I have, that works for me. It also has a sysv init script. Repo is at GitHub, and I also have a brief blog post with links to other possible solutions I found.

There can only be one daemon process running: that is managed by the PID lock file, like most other Linux daemons. To stop it, do

kill `cat /var/run/eg_daemon.pid`

To see if it is running:

ps -elf | grep `cat /var/run/eg_daemon.pid`

Using the pidfile submodule, the PID file is managed automatically. When the daemon is stopped, the pidfile is cleared up. Please see the linked GitHub repo for the init script.

Here's the Python daemon code:

#!/usr/bin/env python3.5
import sys
import os
import time
import argparse
import logging
import daemon
from daemon import pidfile

debug_p = False

def do_something(logf):
    ### This does the "work" of the daemon

    logger = logging.getLogger('eg_daemon')
    logger.setLevel(logging.INFO)

    fh = logging.FileHandler(logf)
    fh.setLevel(logging.INFO)

    formatstr = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
    formatter = logging.Formatter(formatstr)

    fh.setFormatter(formatter)

    logger.addHandler(fh)

    while True:
        logger.debug("this is a DEBUG message")
        logger.info("this is an INFO message")
        logger.error("this is an ERROR message")
        time.sleep(5)


def start_daemon(pidf, logf):
    ### This launches the daemon in its context

    ### XXX pidfile is a context
    with daemon.DaemonContext(
        working_directory='/var/lib/eg_daemon',
        umask=0o002,
        pidfile=pidfile.TimeoutPIDLockFile(pidf),
        ) as context:
        do_something(logf)


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Example daemon in Python")
    parser.add_argument('-p', '--pid-file', default='/var/run/eg_daemon.pid')
    parser.add_argument('-l', '--log-file', default='/var/log/eg_daemon.log')

    args = parser.parse_args()

    start_daemon(pidf=args.pid_file, logf=args.log_file)

For completeness' sake, here is the init script. Note that "kill" is really just a method for sending a POSIX signal -- see man page for signal(7) for an overview. The python-daemon context will catch the signal, terminate the process cleanly closing file descriptors, and delete the PID file automatically. So, it really is a clean termination.

You can write your code to catch SIGUSR1 or something similar, in order to do a reload of the daemon config. There is no advantage to writing Python stop the daemon.

#!/bin/bash
#
# eg_daemon      Startup script for eg_daemon
#
# chkconfig: - 87 12
# description: eg_daemon is a dummy Python-based daemon
# config: /etc/eg_daemon/eg_daemon.conf
# config: /etc/sysconfig/eg_daemon
# pidfile: /var/run/eg_daemon.pid
#
### BEGIN INIT INFO
# Provides: eg_daemon
# Required-Start: $local_fs
# Required-Stop: $local_fs
# Short-Description: start and stop eg_daemon server
# Description: eg_daemon is a dummy Python-based daemon
### END INIT INFO

# Source function library.
. /etc/rc.d/init.d/functions

if [ -f /etc/sysconfig/eg_daemon ]; then
        . /etc/sysconfig/eg_daemon
fi

eg_daemon=/var/lib/eg_daemon/eg_daemon.py
prog=eg_daemon
pidfile=${PIDFILE-/var/run/eg_daemon.pid}
logfile=${LOGFILE-/var/log/eg_daemon.log}
RETVAL=0

OPTIONS=""

start() {
        echo -n $"Starting $prog: "

        if [[ -f ${pidfile} ]] ; then
            pid=$( cat $pidfile  )
            isrunning=$( ps -elf | grep  $pid | grep $prog | grep -v grep )

            if [[ -n ${isrunning} ]] ; then
                echo $"$prog already running"
                return 0
            fi
        fi
        $eg_daemon -p $pidfile -l $logfile $OPTIONS
        RETVAL=$?
        [ $RETVAL = 0 ] && success || failure
        echo
        return $RETVAL
}

stop() {
    if [[ -f ${pidfile} ]] ; then
        pid=$( cat $pidfile )
        isrunning=$( ps -elf | grep $pid | grep $prog | grep -v grep | awk '{print $4}' )

        if [[ ${isrunning} -eq ${pid} ]] ; then
            echo -n $"Stopping $prog: "
            kill $pid
        else
            echo -n $"Stopping $prog: "
            success
        fi
        RETVAL=$?
    fi
    echo
    return $RETVAL
}

reload() {
    echo -n $"Reloading $prog: "
    echo
}

# See how we were called.
case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  status)
    status -p $pidfile $eg_daemon
    RETVAL=$?
    ;;
  restart)
    stop
    start
    ;;
  force-reload|reload)
    reload
    ;;
  *)
    echo $"Usage: $prog {start|stop|restart|force-reload|reload|status}"
    RETVAL=2
esac

exit $RETVAL

Solution 2

A full example is available here.

You should be able to better understand the inner workings of python-daemon.

Moreover the code provided also gives an example of an init script to simply start/stop the daemon. However, you can start/stop it simply by calling the original function again with the argument stop:

python original_func.py stop

Solution 3

As you can see in the 'with' statement documentation, it statement does perform some 'magic', which is related to our purpose. Specifically:

The execution of the with statement with one “item” proceeds as follows:

1. The context expression (the expression given in the with_item) is evaluated to obtain a context manager.

2. The context manager’s __exit__() is loaded for later use.

3. he context manager’s __enter__() method is invoked.

4. If a target was included in the with statement, the return value from __enter__() is assigned to it.

5. The suite is executed.

6. The context manager’s __exit__() method is invoked. If an exception caused the suite to be exited, its type, value, and traceback are passed as arguments to __exit__(). Otherwise, three None arguments are supplied.

What does this mean? If you look closely to the PEP in question, which serves as python-daemon documentation as well (and which indeed could be much improved), you'll see that it implements __enter__() and __exit__():

The class also implements the context manager protocol via __enter__ and __exit__ methods.

__enter__()

Call the instance's open() method, then return the instance.

__exit__(exc_type, exc_value, exc_traceback)

Call the instance's close() method, then return True if the exception was handled or False if it was not.

In other words, open() is not needed, the example given in the PEP (though not explained correctly) works as is. While the with statement does mean something, it does not keep any loop, once the end of its scope is reached, it calls exit(), which in python-daemon means close(). Therefore, you need to put there a while True or which ever infinite loop you consider.

On behave of your second script not working, I can't really tell you, I'm surprised the first already works. If your daemon is stopping, there is a problem with your scripts for sure, you can check your consumerDaemonLogFile. (as a side note, you have a typo 'sderr' --> 'stderr')

Also, you can see in the PEP that if not specified, the working directory property defaults to '/'. this might be the source of your problem if you are using relative paths in your scripts.

Finally, about the last question, you can easily kill you're daemon finding its PID:

ps ax | grep startConsumerDaemons.py

and sending it a SIGTERM:

kill <pid>

The answer provided by gromain does provide a more handy way to start and stop it, with 'daemon.runner()', but it's far more complicated to setup.

Solution 4

here's 2020 python-daemon boilerplate with argparse and signal handling. For a version with logging please visit here.

import os, sys
import datetime
import time
import daemon
import daemon.pidfile
import argparse
import signal
import logging

PROGNAME = 'monitor'
PATHCTRL = '/tmp/' #path to control files pid and lock
pidpath = os.path.join(PATHCTRL,  PROGNAME + ".pid")
parser = argparse.ArgumentParser(prog = PROGNAME)

sp = parser.add_subparsers()
sp_start = sp.add_parser('start', help='Starts %(prog)s daemon')
sp_stop = sp.add_parser('stop', help='Stops %(prog)s daemon')
sp_status = sp.add_parser('status', help='Show the status of %(prog)s daemon')
sp_restart = sp.add_parser('restart', help='Restarts %(prog)s daemon')
sp_debug = sp.add_parser('debug', help='Starts %(prog)s daemon in debug mode')
sp_start.add_argument('-v', '--verbose', action='store_true', help='log extra informations')
sp_debug.add_argument('-v', '--verbose', action='store_true', help='log extra informations')

class MainCtrl:
  thread_continue = True
  thread_token = "token"

mainctrl = MainCtrl()

def main_thread_stop(signum=None, frame=None):
    mainctrl.thread_continue = False
    mainctrl.thread_token = "test"
 
def main_thread(args, mainctrl):
    verbose = False
    if hasattr(args, 'mainctrl'):
      verbose = args.verbose
    if verbose:
      print("MAIN:{0}".format(args))
    try:
      while mainctrl.thread_continue:
         if verbose:
            print("TOKEN:{0}".format(mainctrl.thread_token))
         time.sleep(1)
    except KeyboardInterrupt as ke:
      if verbose:
        print("INFO: Existing...") 
    except Exception as e:
      if verbose:
        print("ERROR: Exception:{0}".format(str(e)))    

def daemon_start(args=None):
    print("INFO: Daemon Start")
    if os.path.exists(pidpath):
      print("INFO: Daemon already running (according to {0}).".format(pidpath))
      sys.exit(1)
    else:
      with daemon.DaemonContext(
        stdout=sys.stdout,
        stderr=sys.stderr,
        signal_map={
            signal.SIGTERM: main_thread_stop,
            signal.SIGTSTP: main_thread_stop,
            signal.SIGINT: main_thread_stop,
            #signal.SIGKILL: daemon_stop, #SIGKILL is an Invalid argument
            signal.SIGUSR1: daemon_status,
            signal.SIGUSR2: daemon_status,
          },
          pidfile = daemon.pidfile.PIDLockFile(pidpath)
          ):
        print("INFO: Running daemon...")
        main_thread(args, mainctrl)

def daemon_restart(args):
    print("INFO: Daemon Restart")
    daemon_stop()
    time.sleep(1)
    daemon_start(args)

def daemon_stop(args=None):
    print("INFO: Daemon Stop {0}".format(args))
    if os.path.exists(pidpath):
      with open(pidpath) as pid:
        try:
          os.kill(int(pid.readline()), signal.SIGINT)
        except ProcessLookupError as ple:
          os.remove(pidpath)
          print("ERROR: {0}".format(ple))
    else:
      print("INFO: process isn't running (according to the absence of {0}).".format(pidpath))

def daemon_debug(args):
    print("INFO: Daemon debug")
    main_thread(args, mainctrl)

def daemon_status(args):
    print("INFO: Daemon Status")
    if os.path.exists(pidpath):
      print("INFO: Daemon is running")
    else:
      print("INFO: Daemon is not running.")

sp_stop.set_defaults(callback=daemon_stop)
sp_status.set_defaults(callback=daemon_status)
sp_start.set_defaults(callback=daemon_start)
sp_restart.set_defaults(callback=daemon_restart)
sp_debug.set_defaults(callback=daemon_debug)

args = parser.parse_args()

if hasattr(args, 'callback'):
  args.callback(args)
else:
  parser.print_help()

Solution 5

A useful documentation is still missing for the module "python-daemon". I personally gave up about using it, and now I successfully use Sander Marechal's daemon code referenced in this answer.

I slightly modified it in order to be able to do things when you call python testdaemon.py stop. Edit 2020/05/07: modified and working with Python 3.

Here is the code.

---

Sample usage:

import sys, daemon, time

class testdaemon(daemon.Daemon):
    def run(self):
        self.i = 0
        with open('test1.txt', 'w') as f:
            f.write(str(self.i))
        while True:
            self.i += 1
            time.sleep(1)

    def quit(self):
        with open('test2.txt', 'w') as f:
            f.write(str(self.i))

daemon = testdaemon()

if 'start' == sys.argv[1]: 
    daemon.start()
elif 'stop' == sys.argv[1]: 
    daemon.stop()
elif 'restart' == sys.argv[1]: 
    daemon.restart()

View more solutions

Comments

[mgibsonbr]

What is the last with supposed to do? (the modules will run when they are first imported; unless I'm missing something, those last statements will do nothing) And what do you mean by "run the first script but not the second"? If no exception was raised then I'm pretty confident that both ran successfully.

[CQP]

Don't know what you mean by "last" with (there is only one), but in the example code for python-daemon, he puts the script to execute as a daemon inside a with statement like that (python.org/dev/peps/pep-3143). The first script executes and is able to receive / process messages (it's a consumer on a an MQ broker). Because the first is on an infinite loop perhaps the startConsumerDaemons script never gets to the second one?

[mgibsonbr]

The example you posted imports some methods from another file (initial_program_setup, do_main_program, etc) and calls them (do_main_program()), one of them within that with statement. Unless there's some obscure feature of Python I don't know of, I'm confident those statements will do nothing. Anyway, as far as I could understand, python-daemon just turns the currently executing program into a UNIX daemon process, it doesn't create new processes or new threads. If one part of the program enters infinite loop, the subsequent parts won't run at all.

[Nikhil Sahu]

Consider using supervisord.org if u want to daemonize python process. python-daemon has bad documentation and has lost development. Also, with supervisor, you dont have to modify any existing code.

[Basj]

Well, the stop method with kill is not very nice... Woulnd't it be possible to it with python mydaemon.py stop, etc.? Moreover we could simplify your example to keep only the core of the question about start/stop daemon etc.

[phzx_munki]

It's a standard Linux way of handling daemons. There is no interactive process. The only way to communicate would be using Linux signals. Using some Python method would mean wrapping up a call to kill.

[alper]

by using the lockfile how can I stop the daemon using daemon.DaemonContext? @bignose

[alper]

I am having following error: AttributeError: module 'daemon' has no attribute 'Daemon'

[alper]

On your python code how can we stop the python process?

[phzx_munki]

The standard way in Linux: service mydaemon stop