How to make cronjobs high available?

Question

What would be a good solution to make a failover pair that has multiple cronjobs running avoid running the processes twice?

Either in failover, or by dividing the jobs between them, and of course being able to take over all jobs when one of the nodes fails.

I could write a script for this, but someone must have fixed this already, or is it such an uncommon request?

Solution 1

If you have some kind of cluster solution to provide high availability, it is fairly straightforward to do this.

We set up all cron jobs on both (or all) nodes in a cluster. Each job starts by executing a small script which works out if this is the master node in the cluster or not (by checking for the cluster floating IP address). If this is not the master node, the check_for_master script exits with an error which causes the whole cron job to fail. If this node is the master, the check_for_master script runs the job as normal.

The contents of the check_for_master script really depend on which cluster software you are using and the OS you are running.

For example, here's a sample crontab entry:

00 04 * * * /usr/local/bin/check_for_master /usr/local/bin/program-you-want-to-run >/tmp/logfile.out 2>&1

Solution 2

You can use 'rcron' for this particular problem. Rcron offers you with a state file, which simply says "active" or "passive", and if it's active your cron will run on a certain machine. If state file is set to passive it won't run. Simple as that.

Your cron jobs that used to look like:

* * * * *    root    echo "foobar"

will need to be changed to:

* * * * *    root    rcron echo "foobar"

and that's it.

Comments

[skinp]

To add to the question, what if the master fails during the execution of a job? Is there any way for the slave to notice it and reexecute it?

[user2989902]

Unfortunately I don't have the luxury of changing the system around, but I will keep this in mind when a new cluster is setup for this purpose.