rsync a list of directories with absolute path in text file

Question

I have a text file containing a list of directories with its absolute path

$ cat DirectoriesToCopy.txt

/data/Dir1
/data/Dir2

I want to use rsync to copy all these directories preserving its absolute path to another location.

I tried the following rsync command, but it doesn't work

rsync -avr --include-from=DirectoriesToCopy.txt --exclude='*/' --exclude='/*'  /  /media/MyDestination/

What is going wrong here?

Solution 1

Use the following command:

rsync -av --include-from=DirectoriesToCopy.txt --include /data/ --exclude='/data/*' --exclude='/*/' / /media/MyDestination/

You need to include /data/ explicitly, you could also have added that to the list in the file. Then exclude all other directories (order is important with includes/excludes).

Note that your usage of -r was redundant as that's included in -a.

EDIT: You could also accomplish the same result with:

rsync -av --relative /data/Dir1 /data/Dir2 /media/MyDestination/

It's not rsync that's forcing you to do difficult things just to copy a couple of directories, it just gives you multiple ways of doing the same thing; in some cases going the include/exclude way may be more suited, here I'd do the --relative thing above (without --relative you'd end up with /media/MyDestination/Dir1 and /media/MyDestination/Dir2, with the --relative the whole source path is copied to the destination).

Solution 2

I want to use rsync to copy all these directories [from a list] preserving its absolute path to another location

Don't use --include or --filter variations, as that will just confuse things. Instead, use --files-from=_filename_. (If you do, make sure you do lots of testing.)

1. Create the directory list in a file, one directory/file per line.

2. Use rsync's --files-from= with the above file.

3. Use --relative / -R option to make sure the source pathnames are copied at the end of the destination.

4. Even if you have the -a option, also include -r. From the man page:

   In both cases, if the -r option was enabled, that dir's entire hierarchy would also be transferred (keep in mind that -r needs to be specified explicitly with --files-from, since it is not implied by -a).

Complete command:

rsync ${DEBUG:+-nv} -arR --files-from=<list_of_files.txt> <top-level-dir>  <target-dir>

(the files in list_of_files.txt must be relative or found in top-level-dir)

(if DEBUG is set, rsync merely prints out what might have been copied.)

Solution 3

The rsync manual warns about this scenario (section “Include/exclude pattern rules”):

this won't work:

+ /some/path/this-file-will-not-be-found
+ /file-is-included
- *

This fails because the parent directory "some" is excluded by the '*' rule, so rsync never visits any of the files in the "some" or "some/path" directories. One solution is to ask for all directories in the hierarchy to be included by using a single rule: "+ */" (put it somewhere before the "- *" rule), and perhaps use the --prune-empty-dirs option. Another solution is to add specific include rules for all the parent dirs that need to be visited. For instance, this set of rules works fine:

+ /some/
+ /some/path/
+ /some/path/this-file-is-found
+ /file-also-included
- *

In your case, I think the simplest approach would be to preprocess the list of directories to include so that whenever you include /path/to/foo, you also include all the parent directories (/path/to, /path, /), and also include subdirectories of the original directories (/path/to/foo/***), and after all this have a rule that excludes everything not previously listed (*).

<DirectoriesToCopy.txt awk '
    {print "+ " $0 "/***"; while (sub(/\/+[^\/]+\/*$/, "/")) print "+ " $0}
    END {print "- *"}
' >rsync-rules.txt
rsync -avr --include-from=rsync-rules.txt  /  /media/MyDestination/

Comments

[Admin]

Do you want to transfer /data/Dir1 and /data/Dir2 including their contents to /media/MyDestination/Dir1 and /media/MyDestination/Dir2 ? Or do you want to include the /data part? How large is this list, is it too big to simply list those entries on the command line? Anyway, your --exclude statements prevent the recursion.

[Admin]

I would like to keep my full path like /media/MyDestination/data/Dir2/ . I added the --exclude to prevent other directories and files in / from getting copied.

[Admin]

In cases where the list of directories are not very large, i guess the following option is more easier rsync -av $(cat DirectotiesToCopy.txt) /media/MyDestination/

[indiajoe]

Thank you that worked. I am a little surprised that one has to define such careful sequence of filters for simply copying a list of directories using rsync. For example, if my list of directories were not under one common directory /data/ but was instead spread around many locations, the filters to define becomes quite complicated.

[indiajoe]

Thanks for the detailed explanation and script to create the rules file. I understand the issue now. I still feel it would have been nice for rsync to have an option like --dirs-from=DirectoriesToCopy.txt as a short cut to all the complicated filters.

[wurtel]

See my edit to my answer.

[indiajoe]

Thanks. So if my list of directories is not very huge, I could use the --relative flag as shown below. rsync -av --relative $(cat DirectotiesToCopy.txt) /media/MyDestination/

[wurtel]

Yes; keep in mind --relative is just another option.

[Adambean]

This is what I needed! However "--include-from" should be "--files-from". Also I used "--list-only" to see a preview of what RSYNC will do before actually letting it work.

[Otheus]

Thanks, I've updated the answer. --include-from will also work, but then what I said about -r is largely irrelevant.

[leroyse]

I would use $(cat DirectoriesToCopy.txt | xargs) instead to make sure the arguments that are one on each line are actually passed as arguments to rsync

[wurtel]

Whether the arguments are on a separate line or not makes no difference whether you do $(cat DirsToCopy.txt) or $(cat DirsToCopy.txt | xargs), the shell interprets all white space as equal (see $IFS) . Also lookup OOUC ;-)