How can I trigger a script to run after the rsyncdaemon received file changes to a certain folder?

Question

I have several servers where rsyncd is running. I don't use an ssh tunnel, but the native rsync protocol. One folder of the synchronized files contains the files that should form an iso image. Whenever a file in that folder is uploaded (pushed from remote to this server), it shall recreate the iso file automatically. Ideally only file content or size changes should trigger this because I want to show the date of the last change of the iso inside a webpage where the users can download and burn it.

Because the iso is large I want to create it on each server if just one file changes. I don't want to sync the iso itself

Solution 1

This is more a response to Gilles but I wanted to format a little code; a slick way you can do the monitoring of the changed files is to store a md5sum of md5sums of the directory, ala:

find /path/to/iso/data/ -type f -exec md5sum {} + | awk '{print $1}' | sort | md5sum | awk '{print $1}'

...which gives you output of the type:

b843afc89097f0abc062e0d0e14d480b

If you save that on each iteration of your cron, it's a pretty quick and effective way to determine if contents have changed on the target. You could even build a wrapper around the xinetd rsync daemon; replace the call to the binary in /etc/xinetd.d/rsync:

server          = /usr/bin/rsync

...with a script:

server          = /path/to/script.sh

And then in your script compare and do the deed every time the daemon exits with a valid 0 status. That's way more automated than a cron job, with the benefit it only runs if you code it to rsync exit 0. Something like... (psuedo code here):

#!/bin/sh

/usr/bin/rsync "$*"

if [ $? -eq 0 ]; then
  OLDVALUE=`cat /var/cache/isodata.txt`
  NEWVALUE=`find /path/to/iso/data/ -type f -exec md5sum {} + | awk '{print $1}' | sort | md5sum | awk '{print $1}'`
  if [ ${OLDVALUE} != ${NEWVALUE} ]; then
    -- run ISO making code --
    echo ${NEWVALUE} > /var/cache/isodata.txt
  fi
fi

That's the general idea to start with.

Solution 2

The old-fashioned, portable, clumsy way is to run the iso generation script at regular intervals, either via cron or with a while sleep loop (preferable if the interval is not markedly larger than the time it takes to regenerate the iso). To avoid regenerating the iso if the files haven't changed, do it only if [ -n "$(find -cnewer foo.iso -print -quit)" ]. Note that this will not trigger if some files were deleted and none were changed or added.

A better, but non-portable way is to use your platform's file change monitoring API: fam on IRIX and some other systems, kqueue on *BSD, inotify on Linux, etc. All can be easily set up to run a command if a file changes (including a directory); I don't know if they can all watch a directory tree.

A different way would be to put the files on a userland filesystem (such as FUSE) that just passes on calls to the underlying storage except that modifications trigger the regeneation of the iso.

Whatever you end up doing, note that a file could change while you're regenerating the iso; in that case you have two strategies. You can wait until the generation is finished and start again, which is wasteful. Or you can kill the iso generation and start again, but that could lead to a long time without an iso update if files keep being modified faster than the iso can be regenerated. A mix of the two where you kill the iso generation unless no iso has been generated in more than X time is a reasonable solution.

Comments

[Christian]

Thank you very much, both things that you suggest work great together! I would left out the "awk '{print $1}'" after md5sum, because by leaving it in you basically guarantee that renames won't be detected, only content changes or adding/removing files