Does anyone know why rsync would keep sending the files over and over again?

Question

I'm trying to using rsync to backup some files, about half a TB. It's now it a state where it keeps sending the same files everytime it runs.

for example:

rsync -av /data/source/* user@host:/data/dest
sending incremental file list
source/file1.txt
source/file2.txt

I then verify those files are copied over... then the next time it runs it does the same thing

rsync -av /data/source/* user@host:/data/dest
sending incremental file list
source/file1.txt
source/file2.txt

any idea why it's getting stuck on these files? I've tried to wipe the whole dest directory out and start over but no luck.

thanks,

Solution 1

Use --itemize-changes to get rsync to output what is actually being changed

Options that affect the decision for whether or not to skip files are:

• -c which decides whether to skip identical files based on checksum
• -I which ignores size and time when deciding whether to skip files.
• -t which is used to preserve modification times (also turned on by -a flag). This option is required if you want to use modification times to decide whether or not to compare files based on modification time, and so it is required if you want to not evaluate files that have the same modification time (otherwise it will be as though you used -I).

Although rsync may be sending the files again, it shouldn't be transferring all the contents if they haven't changed - running with -v should print a summary of how much data was matched in the transfer.

For checking, the following should help:

• md5sum of the files on either end - to show you if the contents have changed
• ls -l should show you if the timestamps have changed.

Solution 2

I remember a similar problem with two systems' clocks not quite behaving. I had to use --modify-window=60 to account for "temporal anomalies".

Comments

[Admin]

Have you done a diff on the files, and checked all the information on their ls -l list? They could, in principle, be modified at either edn without you being aware of it, and maybe rsync doesn't leave the files in the right state on the destination.

[AdishRao]

But -t has anyway an effect on file transfer. From rsync man: Note that if this option (-t) is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing -t or -a will cause the next transfer to behave as if it used -I

[AdishRao]

It helps also with --modify-window=1 when transfering on FAT filesystems as FAT stores the times with a 2-second resolution, so times can differ of +/- 1s.

[David Fraser]

Good point, thanks Seki! I've updated the answer