rsync: --include-from vs. --exclude-from what is the actual difference?

Question

In the documentation, it mentions these as being files containing lists of either patterns to include or patterns to exclude. However, that implies for inclusions, everything is considered an exclusion except where things match patterns. So for example, an include file containing:

/opt/**.cfg

Should only include any file named *.cfg that exists anywhere under a directory named opt any where in the tree. So it would match the following:

/opt/etc/myfile.cfg
/some/dir/opt/myfile.cfg
/notopt/opt/some/other/dir/myfile.cfg

I'd therefore expect it to implicitly exclude anything else. But that doesn't seem to be the case, since I am seeing this in the itemized output:

*deleting   etc/rc.d/init.d/somescript

So what is the deal with --include-from and --exclude-from? Are they just aliases for --filter-from?

Solution 1

rsync doesn't work like that. Any file with a filename pattern that does not match any of the include or exclude patterns are considered to be included. In other words, think of the include pattern as a way of overriding exclude pattern.

From the docs (emphasis mine):

Rsync builds an ordered list of include/exclude options as specified on the command line. Rsync checks each file and directory name against each exclude/include pattern in turn. The first matching pattern is acted on. If it is an exclude pattern, then that file is skipped. If it is an include pattern then that filename is not skipped. If no matching include/exclude pattern is found then the filename is not skipped.

So, if you want to include only specific files, you first need to include those specific files, then exclude all other files:

--include="*/" --include="*.cfg" --exclude="*"

Couple of things to note here:

1. The include patterns have to come before the excludes, because the first pattern that matches is the one that gets considered. If the file name matches the exclude pattern first, it gets excluded.

2. You need to either include all subdirectories individually, like --include="/opt" --include="/opt/dir1" etc. for all subdirectories, or use --include="*/" to include all directories (not files). I went with the second option for brevity.

It is quirky and not very intuitive. So read the docs carefully (the "EXCLUDE PATTERNS" section in the link) and use the --dry-run or -n option to make sure it is going to do what you think it should do.

Solution 2

If you (like me) have a hard time to wrap your head around the FILTER RULES-section in the man-pages but have a basic understanding of find, you could use that instead.

Say you whant to sync everyting with a specific date (ex 2016-02-01) in either the file-name or in a directory-name from /storage/data to rsync_test. Do something like this:

cd /storage/data
find . -name '*2016-02-01*' \
  | rsync --dry-run -arv --files-from=- /storage/data /tmp/rsync_test

Comments

[Craig]

That makes sense. I am actually using this to verify there are no ambiguous files outside the scope of an exclude and include list. I first run with everything excluded and only explicit inclusions being synchronised. I then run the same command, this time with everything included and only explicit exclusions not being synchronised. The itemized logs are then diffed and any ambiguous files not matched by either pattern will be revealed. This ensures nothing is modified unexpectedly.

[Hari Menon]

Actually, even my sample wouldn't work. Because exclude=* excludes everything, and that is the first matching pattern, so nothing gets transferred. I have changed it to have the include patterns first.

[Craig]

In the end --filter="merge exclude_list" and --filter="merge include_list" was sufficient. I created an implicit rule at the bottom of each to exclude/include anything not matched. It now behaves as expected. Thanks!

[Max]

This is the best explanation of the weird rsync rules that I've seen so far.

[John Tyree]

Don't forget -m to avoid creating huge trees of empty directories.

[Matthias M]

Also have a look at this similar question askubuntu.com/questions/56401/…

[Frank Nocke]

@hari-shankar, you did adjust the code, but not the explaining sentence right before it. I just did that now.