How to feed grep output into scp?

Question

I have a file where each line is a file name. Content of the file:

file1
file2
file3
...

I then run a grep command to get a subset of the list of the file names.

How do I feed the output of the grep command into scp? I have played around with xargs and $() without much success...

Solution 1

No need to use xargs:

for file in $(grep <some-pattern> <filelist>); do scp $file <remote>; done;

Where

• <some-pattern> is your grep pattern
• <filelist> is your list of files
• <remote> is your remote end of the scp command

The for loop will iterate over each line produced by grep and pass this to $file, which you can re-use in the commands executed between do and done.

Also see: Advanced Bash-Scripting Guide: – Loops

Solution 2

To iterate over a list of file names with one name per line, you have several possibilities:

• Use command substitution to work the file into a list of file names inside the shell. You need to take precautions, because you can't put the command substitution in double quotes, since the result of the command substitution needs to be split into words. You need to turn off globbing and set IFS to only split words on newlines.

  set -f; IFS='
  '
  scp -- $(…) remote.example.com:
  

• Use the read built-in to process the file names in a loop. Again, you need to be careful, because read treats backslashes and $IFS characters specially.

  … | while IFS= read -r x; do scp -- "$x" remote.example.com: done
  

• You can use GNU xargs, but you have to tread carefully, because xargs expects its input in an oddly quoted format. You must pass the -d '\n' option to explicitly select newlines as the input separator and turn off the special behavior of whitespace and \'".

  … | xargs -d '\n' -I {} scp {} remote.example.com:
  

  The only advantage of xargs is that it can group multiple arguments onto one invocation up to the command line length limit, but that only works if the arguments are passed last to the command. You can achieve that through an intermediate call to sh, but it's rarely worth the trouble.

  … | xargs -d '\n' sh -c 'scp -- "$@" "$0"' remote.example.com:
  

• An easier way to use xargs is to get a null-separated list instead of a newline-separated. This requires the -0 option, which is supported by GNU xargs and also recent *BSD. As above, shoving the arguments into a non-final position requires a bit of work.

  … | tr '\n' '\0' | xargs -0 sh -c 'scp -- "$@" "$0"' remote.example.com:
  

The … part would be grep NAME_REGEXP /path/to/filenames.list if you meant to filter the file names with a regular expression. If you meant to filter on the file contents, use one of the techniques above to feed the file names to grep, e.g.

set -f; IFS='
'
scp -- $(grep -l CONTENT_REGEXP -- $(cat /path/to/filenames.list)) \
    remote.example.com:

Another possibility that bypasses the shell difficulties is to use rsync and its --files-from option. Assuming the file names are all relative to the current directory:

grep NAME_REGEXP /path/to/filenames.list |
rsync -a --files-from - . remote.example.com:

Comments

[SamK]

Dont forget to quote your filenames (in case of spaces).

[Gilles 'SO- stop being evil']

If any of the file names contains whitespace or globbing characters, this won't work.

[slhck]

Very elaborate, I'll give you that! (Just wondering, does that still justify downvoting if it worked for the OP?)

[Gilles 'SO- stop being evil']

@slhck Any answer that's wrong (and I don't just mean wrong to detail, if it had been just a matter of adding a quote or two I'd just have edited your answer) is a legitimate downvoting target, in fact it should be downvoted. An answer that looks right when you test it on a simple example but breaks in real-world scenarios is the worst kind.

[slhck]

Well, imo it's not "wrong" when it does work -- even if it might be a simple scenario. The button actually says, "This answer is not useful", and it obviously was useful. I knew it wasn't gonna work with complex file names, and by adding your comment above, that was absolutely clarified. But yeah, let's leave it at that, not going to fight over it!

[dognar]

Thanks for your detailed answer Gilles, I'm sure it will be handy if I come across complex filenames.

[StatsSorceress]

Now suppose there are thousands of files to search through. What is an efficient way?

[slhck]

@StatsSorceress Faster than grepping through the files? Can't think of anything now, but Gilles' answer is better in any case.