Getting notified of failed cron jobs

Question

I'm concerned that cron jobs can fail silently for an indefinite period of time on vanilla Ubuntu Desktop 12.04.1 (Precise), and no one will notice. I would like to get a notification whenever a system cron job prints some output or just fails.

I know it is possible to install a mail server (e.g. postfix), configure it for local-only delivery, set up an alias so that root's mail is delivered to my normal user account and configure a mail client to check my local mailbox.

Are there any lightweight alternatives to this solution in Ubuntu?

Solution 1

You could redirect the error output of your cronjob command to a file. Here is an example of a line in /etc/crontab:

01 3    * * *   user    /bin/command 2>> /var/log/some.file

Then at least you got a clue if errors occured. You might even write a script to notify you over notify-osd or similar tools when the file changes.

Edit:

The file /var/log/syslog reports messages from cron as well. You might want to take a look at that. To get a dedicated log-file for the cron deamon edit /etc/rsyslog.d/50-default.conf and uncomment/edit the line that says:

#cron.*             /var/log/cron.log

Don't know what you'll find there but worth the try. Report how it went.

Solution 2

Jenkins is a fairly helpful tool for this sort of thing. I know people tend to think of it as a CI, but it's really just a job execution tool. It will capture the output, run time and exit code status of a job which it uses to determine whether or not the job is a failure.

You can setup ssh-agent to have Jenkins connect to a remote machine, schedule jobs to run just like crons, automatically space them based on run time (instead of "3am"), chain dependencies, rotate logs of output on a per-job basis and integrate with a bunch of outside systems (Slack, Hipchat, etc) via plugins to be notified of failures.

Last time I set this up, it was a huge help. We knew immediately if there were issues anywhere and were able to centrally control and track crons from multiple different systems.

Solution 3

"I would like to get a notification whenever a system cron job prints some output or just fails."

I'd recommend using some sort of cron monitoring tool. There are a few out there but I currently use Dead Man's Snitch (https://deadmanssnitch.com) and like it. It will alert you when a cron job doesn't check in. Like you're doing, just curl your unique snitch URL after the job and hit the url. There are a few others out there like probyapp but they aren't free...good luck.

Comments

[Till Ulen]

Thanks. I'm worried about existing and future system cron jobs. So I'd rather not modify them all. That would hinder upgrades and would not cover any new cron jobs received as part of a package update. So I'm looking for a solution that would apply to all system cron jobs at once.

[Till Ulen]

Do you have an idea how to integrate such a script with cron? Modifying all system cron jobs one by one is not a good idea. I'm looking for a solution that would apply to all system cron jobs at once.

[con-f-use]

I'd be surprised if a solution existed outside of the usual mailer that fulfilled so many restraints

[Shane H]

I hope you don't mind me saying, I think you should check out cronitor.io. I built Cronitor after trying to use DMS but realizing that it had some real flaws. Basically, it can take up to 29 days to be notified that a monthly cron job failed to run. We built Cronitor with python on ubuntu, you should come check it out.

[Pēteris Caune]

I've built another one: healthchecks.io, code on github

[aklmie]

@Peteris Awesome work

[Akhil]

down voted. sorry, bro.