Rsync --- Itemized List of Changes

Question

I've learned that with the -i option, I can get rsync to list all the changes it makes. I've been using it along with the -n or --dry-run options so that I can always learn about all expected changes prior to actually executing them.

I've been using rsync mainly to sync my home directories on my two computers, both of which are connected to a network. Often times I need to move just a small number of relatively small files. If this is the case, rsync spends more time going through all the files it's going to exclude from the transfer rather than actually transferring the actual changed files.

Now if I follow this procedure where I first do a dry run and list all the changes, and then actually proceed, the longest part of calculating the files that are to be excluded from the transfer gets done two times.

I'd like to cut it to just one. Is there a way to feed the itemized list of changes created by the dry run back to rsync so that the live run is faster or do something to that effect?

Solution

from the man page of rsync:

-F same as --filter='dir-merge /.rsync-filter' repeated: --filter='- .rsync-filter'

        --exclude=PATTERN       exclude files matching PATTERN
        --exclude-from=FILE     read exclude patterns from FILE
        --include=PATTERN       don't exclude files matching PATTERN
        --include-from=FILE     read include patterns from FILE
        --files-from=FILE       read list of source-file names from FILE

so run rsync -i first and output it to a file and use --files-from option or you can use find utility for finding the last modified file and then rsync them. see https://serverfault.com/questions/115945/synchronizing-very-large-folder-structures

Comments

[N.N.]

You may want to try out Unison which, if I understand correctly, will do this in one run.

[Gilles 'SO- stop being evil']

I second N.N.: this sounds like exactly what Unison was designed for. Only use rsync if you're always synchronizing in the same direction, otherwise you will lose work because one day you'll make a mistake deciding which way to synchronize. Unison figures this out automatically and tells you if (and only if) there is a conflict (different changes to the same file on the two computers).

[sr_]

Sounds like a use case of rdup or maybe even git-annex.

[Remon]

look at csync2 and lsyncd if you want some more customization see axivo.com/community/threads/…

[PSkocik]

Thanks. The problem with this approach is that the output of -i doesn't just list the files that are to be transferred/synched, it also lists deletions. If you're running with the backup and backup-dir options, those deletions change to mv's. I suppose I could grep out just actual transfers and then handle deletes separately... I was kind of hoping for a more elegant solution that wouldn't involve me having to write my own wrapper scripts.

[Remon]

you can use excludes with the “perishable” modifier.Recent versions of rsync allow filter rules to be flagged with a perishable modifier. The rsync man page describes this modifier as follows.A p indicates that a rule is perishable, meaning that it is ignored in directories that are being deleted. see blog.mudflatsoftware.com/blog/2012/10/31/…