Solution for running scheduled cron job within a Docker container?

Question

I have a Docker container in which I have my Python tools installed, including my Luigi pipeline interface. I would like to run a shell script which kicks off my Luigi pipeline on a weekly basis using cron.

I have tried high and low to get cron to work within a Docker container. I cannot, for the life of me, get my crontab -e file to run.

In my file I have:

0 0 * * Sun /data/myscript.sh

followed by a new line. Cron is running in the background - ps aux | grep cron shows /usr/sbin/cron is running. Furthermore, in my /var/log/syslog file, I have:

/USR/SBIN/CRON[2037]: (root) CMD (/data/myscript.sh)

I've also tried using 0 0 * * Sun . /root/.bashrc ; sh /data/myscript.sh

However, my script does not run (when I run my script manually using bash myscript.sh, I get the expected results).

Suggestions?

Solution 1

Scheduled tasks won't run inside of a normal container since there is no scheduler running. The only active task will be that which you have elected to run via the CMD keyword or the Entrypoint.

In order to execute schedule tasks, it's more prudent to utilize the host scheduler and docker exec commands:

docker exec <container> <command>
docker exec <container> /data/myscript.sh

So you would end up with a cron on your host something like :

(Crontab Style) 0 * * * * root docker exec mycontainer /data/myscript.sh

If you have a cluster, you would have to query the cluster first to locate the container, or even have a script do it for you.

Solution 2

A container is meant to only one run main process. You either need to run crond as the main process for a container, or ensure that crond is running alongside your main process. This kind of breaks the contracts / point of containers, but sometimes it's easier to set it up this way. Instructions below:

My Dockerfile has the following ENTYPOINT:

ENTRYPOINT ["/startup.sh"]

And then within startup.sh I do a couple of things to spin up the container, but most importantly before executing the last command, I have this:

crond
exec start_my_service

crond starts the daemon that executes the crons, and start_my_service then becomes the primary process for my container.

Comments

[Dockstar]

As a follow up, in a swarm / kubernetes environment, I've found it more useful to write Daemons that are always running, but only execute actions at specified times. This allows your tasks to be distributed rather cleanly within the environment.