Randomizing minute for crontab entries between servers

Question

I have a common puppet recipe for a set of our servers. One of the puppet-managed files is /etc/crontab that contains the original cron.daily, cron.weekly and cron.monthly entries.

The problem of the common /etc/crontab file here is that all cronjobs are run at the exactly same time in all of our servers. For example, daily backups consumed all our backup server resources as all servers are feeding it at same time.

What is the preferred way to randomize the exact minute of the daily/weekly/monthly runs between servers, while still keeping the puppet recipe common between all servers?

I have been thinking a few different options:

• Running sleep & $RANDOM before the actual job. In this way, the crontab would be similar in every host but the exact time of running the daily/weekly jobs would vary. There would be variation even inside the same server (for example daily run interval could vary between 23-25 hours).
• Calculating an integer value from the hostname and running all jobs with at + "the hostname-based integer" minutes. A bit hacky, but might work...
• Some other method, but what?

Solution 1

Puppet has an fqdn_rand function that you can use for this:

minute => fqdn_rand(60)

Solution 2

Instead of randomizing cronjobs, you should randomize your scripts to sleep.

I usually do this kind of thing with:

sleep $((RANDOM%120)) && /bin/sh -c /path/to/my/cronjob

In this way all cronjobs can start at the same time, but they will randomically sleep between 0 and 120.

Solution 3

Yes, you can get "random" integer by hashing FQDN. In erb-template:

<%= Time.at( fqdn.hash % 10000 ).strftime('%M')  %> * * * * myuser /bin/mycommand

But puppet can do it more gracefully. In manifest:

$minute = inline_template("<%= Time.at( fqdn.hash % 10000 ).strftime('%M')  %>")
cron { my-cron:
    minute => $minute,
    ...
}

Solution 4

Here is a lot of different kind of implementation examples : http://projects.puppetlabs.com/projects/1/wiki/Cron_Patterns

View more solutions

Comments

[Luke404]

I don't like this solution, because just looking at cron logs you have no way to know how much time has been spent sleeping and how much executing the script, you need additional logging from your script and/or the processes you are executing.

[Ville Mattila]

Simple as that! I had completely missed this function. Excellent, thanks!

[Marcel]

cron logs don't log the time of execution? i doubt that. you can always use logger -p info to log everything using the already available syslog.