Cron and virtualenv

Question

I am trying to run a Django management command from cron. I am using virtualenv to keep my project sandboxed.

I have seen examples here and elsewhere that show running management commands from within virtualenv's like:

0 3 * * * source /home/user/project/env/bin/activate && /home/user/project/manage.py command arg

However, even though syslog shows an entry when the task should have started, this task never actually runs (the log file for the script is empty). If I run the line manually from the shell, it works as expected.

The only way I can currently get the command to run via cron, is to break the commands up and put them in a dumb bash wrapper script:

#!/bin/sh
source /home/user/project/env/bin/activate
cd /home/user/project/
./manage.py command arg

EDIT:

ars came up with a working combination of commands:

0 3 * * * cd /home/user/project && /home/user/project/env/bin/python /home/user/project/manage.py command arg

At least in my case, invoking the activate script for the virtualenv did nothing. This works, so on with the show.

Solution 1

You should be able to do this by using the python in your virtual environment:

/home/my/virtual/bin/python /home/my/project/manage.py command arg

EDIT: If your django project isn't in the PYTHONPATH, then you'll need to switch to the right directory:

cd /home/my/project && /home/my/virtual/bin/python ...

You can also try to log the failure from cron:

cd /home/my/project && /home/my/virtual/bin/python /home/my/project/manage.py > /tmp/cronlog.txt 2>&1

Another thing to try is to make the same change in your manage.py script at the very top:

#!/home/my/virtual/bin/python

Solution 2

Running source from a cronfile won't work as cron uses /bin/sh as its default shell, which doesn't support source. You need to set the SHELL environment variable to be /bin/bash:

SHELL=/bin/bash
*/10 * * * * root source /path/to/virtualenv/bin/activate && /path/to/build/manage.py some_command > /dev/null

It's tricky to spot why this fails as /var/log/syslog doesn't log the error details. Best to alias yourself to root so you get emailed with any cron errors. Simply add yourself to /etc/aliases and run sendmail -bi.

More info here: http://codeinthehole.com/archives/43-Running-django-cronjobs-within-a-virtualenv.html

the link above is changed to: https://codeinthehole.com/tips/running-django-cronjobs-within-a-virtualenv/

Solution 3

Don't look any further:

0 3 * * * /usr/bin/env bash -c 'cd /home/user/project && source /home/user/project/env/bin/activate && ./manage.py command arg' > /dev/null 2>&1

Generic approach:

* * * * * /usr/bin/env bash -c 'YOUR_COMMAND_HERE' > /dev/null 2>&1

The beauty about this is you DO NOT need to change the SHELL variable for crontab from sh to bash

Solution 4

The only correct way to run python cron jobs when using a virtualenv is to activate the environment and then execute the environment's python to run your code.

One way to do this is use virtualenv's activate_this in your python script, see: http://virtualenv.readthedocs.org/en/latest/userguide.html#using-virtualenv-without-bin-python

Another solution is echoing the complete command including activating the environment and piping it into /bin/bash. Consider this for your /etc/crontab:

***** root echo 'source /env/bin/activate; python /your/script' | /bin/bash

Solution 5

I am sorry for that nth answer but I checked the answers and there is really simpler and neater.

Long story short

Use the python binary of your venv in your cron :

0 3 * * * /home/user/project/env/bin/python /home/user/project/manage.py 

Long story

We activate the virtual environment when we want to set the current shell with the python config of that specific virtual environment(that is binaries and modules of that).
It is relevant to work with the current shell : execute multiple python commands on the current shell without the need to reference the full python path of the venv.
In the frame of a cron or even a bash, which value to activate the environment ? Besides I read in some answers some references to bash rather than sh or still to define a wrapper to call the Python code. But why the hell should we bother with these ?

I repeat, just do it :

0 3 * * * /home/user/project/env/bin/python /home/user/project/manage.py 

The documentation confirms that :

You don’t specifically need to activate an environment; activation just prepends the virtual environment’s binary directory to your path, so that “python” invokes the virtual environment’s Python interpreter and you can run installed scripts without having to use their full path. However, all scripts installed in a virtual environment should be runnable without activating it, and run with the virtual environment’s Python automatically.

View more solutions

Comments

[rettops]

One difference that I see is that the script will run manage.py with /home/user/project as the current working directory. Your cron command would be run with your home directory as the cwd. Maybe the log file is there?

[John-Scott]

Actually the log path is defined absolutely, it's simply not created/appended to because the script is not running.

[jberryman]

A quick and dirty solution to cron issues is to dump your environment (in which your command is inexplicably working) with env and export them all in a bash script wrapper you call from the crontab.

[John-Scott]

That also does not work. Forgot to put that in my list of things that do not work. Yes, I can run that command manually in the shell but it does not work from cron.

[ars]

Did you replace ~ with the full path? (You probably did, just making sure ...)

[John-Scott]

Ah, you've come up with a working example! I've tried about every combination and activating the virtualenv appears to have no effect whatsoever. I do set my PYTHONPATH in .bashrc but this apparently is not used by cron? Will update my question to highlight your answer.

[ars]

Yeah, I'd forgotten that cron runs under a very minimal environment. The general recommendation is to write bash scripts to set up whatever environment your job will need. You could try sourcing the bash profile directly in cron, but this can lead to subtle bugs depending on what's in your profile (perhaps if you have a separate and minimal profile for such needs, it would be fine).

[Dick]

A good way to test is to execute /bin/sh, and then try to execute your command from there. At least you'll have the same environment setup as cron.

[Reed Sandberg]

Or '.' (dot command), which is supported by /bin/sh . /path/to/virtualenv/bin/activate

[Aaron Schumacher]

I'm very curious as to whether there's consensus that this is in fact the only correct way.

[Will]

This is probably the only correct way. But there are other ways that works.

[Canucklesandwich]

This isn't "the only correct way." I've successfully executed a script in a virtualenv simply by pointing the cronjob to the virtualenv's python binary, such as '/home/user/folder/env/bin/python'. No need to activate the environment whatsoever.

[varela]

If you use custom PYTHONPATH in virtual environment env/bin/python will not work for you. That's why using of env/bin/activate is better

[Q Caron]

cron jobs may fail because your packages are not installed in your virtualenv. Make sure you are using your virtualenv's pip so installed packages can be reached with the first instruction of this solution. Maybe you will have to call the virtualenv's pip by using its absolute path when installing packages (stackoverflow.com/questions/20952797/…)

[Admin]

it depends on how you set the PYTHONPATH and if you set it in a way that requires "activating" the venv, you're doing it wrong

[joemurphy]

DavidWinterbottom, if that is your real name, you're my hero. I never knew that about sh vs bash and source files. You've shone a light into my little bash-scripting world dude. Thanks.

[Austin]

I also had to explicitly declare my settings since I have different ones for local and productions. --settings=path.to.settings

[dspacejs]

If you have a postactivate file, you should be doing source /path/to/virtualenv/bin/activate && source /path/to/virtualenv/bin/postactivate

[Victor Schröder]

Not a good solution. Every python tasks in the crontab would then run with the binary from the virtualenv. Making that binary a pseudo-global python goes against the very purpose of virtualenv.

[user12345]

Regarding adding yourself to /etc/aliases this is a good read: https://unix.stackexchange.com/questions/65013/understanding‌​-etc-aliases-and-wha‌​t-it-does

[Martin Becker]

Thanks! For me, this works rather than the accepted answer by Gerald.

[adnanmuttaleb]

what is 'root' for? can anybody explain

[jask]

It worked for me. I could not run using absolute PATH with venv. Thanks

[pico_prob]

@adnanmuttaleb, I assume you know the root user and its privileges (king of the kings). The first column after cronjob timing is the user to execute the cronjob.

[pico_prob]

'Don't look any further' is a bold statement. Particularly, when the following answer claims to know the 'only correct way': stackoverflow.com/a/22724628/6919635

[wisbucky]

bash -c is unnecessary. Just use . command instead of source. You won't need to change the SHELL variable in crontab, or wrap your command with bash -c. See stackoverflow.com/questions/3287038/cron-and-virtualenv/…

[wisbucky]

You don't need to pipe the command to bash. Just use . command instead of source. See stackoverflow.com/questions/3287038/cron-and-virtualenv/…

[wisbucky]

You can use ~ in crontab command statements in a user crontab. See stackoverflow.com/questions/3287038/cron-and-virtualenv/…

[Basil Musa]

The whole point of bash -c is to avoid setting the SHELL environment in cron. If you are sure that your scripts won't behave differently in sh then you can go ahead, but if you have tested on bash, then they might behave differently in sh. To stay on the safe side, I would prefer sticking to using bash -c or whatever your default shell is.

[Alex]

Does not work for me. The virtualenv is ignored.

[Alex]

Does not work for me. The venv is completly ignored

[Basil Musa]

For it to work, your command should include the following cd /home/user/project && source /home/user/project/env/bin/activate && ./manage.py command arg

[Gilberto Albino]

It seems like people will never mind about learning the docs for the technologies they work with! Thanks for letting decency still exist in this planet!

[IdoS]

What "command arg" mean here? Could you please share a full example?