Daemonizing a python script in debian using virtualenv

Question

I've seen a lot of scripts for daemonizing a python script in linux, but not much information about how to use them. Could anyone guide me on this?

I currently have a lengthy python script that listens on a socket for an incoming message, if it's the correct format accepts it and then stores it into the database. The script itself just opens the socket and then listens on a while true (which does the job!) and does all the work in there.

To daemonize it, would I have to modify my current script or call it from a separate script? I've seen examples of both but got neither to work.

Also, I'm using virtualenv which might the root of my problems, any hints on using this with daemonized scripts?

Solution 1

Create a shell-script that activates the virtual environment, and runs your Python script in the background.

Also, there should by a python module in the virtual environment that you can import and activate the environment from too. I don't have virtualenv working at the moment, so I can not check where it is, but search for activate (or something similar) in the virtual environment and you should find it.

Edit: Added a minimal Debian init.d script

The absolute minimal script needed to start a daemon when the computer boots, is this:

#!/bin/sh
/path/to/program &

The & makes the program run in the background, so it wont stop the rest of the boot process.

For a more complete script, copy /etc/init.d/skeleton and edit the new file. The important part to edit is the block at the beginning (between ### BEGIN INIT INFO and ### END INIT INFO, which is used by the update-rc.d program), and the NAME, DAEMON and DAEMON_ARGS variables. Hopefully that should be all that's needed for making a startup-script.

Activate the script as this:

sudo update-rc.d <name of script> defaults
sudo update-rc.d <name of script> enable

And to start it:

sudo update-rc.d <name of script> start

The <name of script> is just the name, not the full path.

Solution 2

script
  export PYTHONPATH=.:/home/ubuntu/.local/lib/python2.7/site-packages/:/home/ubuntu/python/lib/python2.7/site-packages/
  exec start-stop-daemon --start  --chuid ubuntu --exec /home/ubuntu/python_envs/MyProj/bin/python /home/ubuntu/www/MyProj/MyProj.py -- --config-file-dir=/home/ubuntu/www/MyProj/config/ >> /home/ubuntu/startup.log 2>&1 &
end script

When you need to run an application in a python virtualenv, you can either 'activate' the virtualenv, or use that environment's unique python command.

As per the website "If you directly run a script or the python interpreter from the virtualenv's bin/ directory (e.g. path/to/env/bin/pip or /path/to/env/bin/python script.py) there's no need for activation." - http://pypi.python.org/pypi/virtualenv

I also have some python modules that were compiled from source. Those need to be in the PYTHONPATH environment variable. That could be part of your virtualenv activation, done with virtualwrapper, or explicitly called (as I do below.)

Calling the program from an UPSTART job works as well. My example is above.

On an Ubuntu 10.10 instance on Amazon EC2, I had better luck with the start-stop-daemon command. I also struggled with some of the other upstart 'stanzas.' I am calling a python application with a specific virtualenv and some parameters to my executed program.

Comments

[ingh.am]

I have the activate file yes, so I write a bash script? How do I daemonize it?

[Some programmer dude]

@ing0: Maybe you mean how to write a script that is started when the computer boots up, like all the other programs in /etc/init.d? Then it depends a little on what Linux distribution you use. Try man update-rc.d, if the manual page exists it should contain pointers on how to do it. Also try google for startup script _linuxdist_.

[ingh.am]

Currently using a variant of debian, all the stuff I've found online provides code like this: code.activestate.com/recipes/… Is this sort of code not required?

[Some programmer dude]

@ing0: I updated my answer with a little description on how to make a boot script.

[ingh.am]

Thanks so much. I'll let you know how I get on! :)

[hobs]

Do you have any example scripts (shell and/or python) that work with upstart? You mention "example is below".

[Alex]

This only starts the daemon at boot. A real daemon process should have the option to start/stop at any time. Sure the sh script could be started (with & ) but then it gets killed when the terminal window closes.