grep files from list

Question

I am trying to run grep against a list of a few hundred files:

$ head -n 3 <(cat files.txt)
admin.php
ajax/accept.php
ajax/add_note.php

However, even though I am grepping for a string that I know is found in the files, the following does not search the files:

$ grep -i 'foo' <(cat files.txt)

$ grep -i 'foo' admin.php
The foo was found

I am familiar with the -f flag which will read the patterns from a file. But how to read the input files?

I had considered the horrible workaround of copying the files to a temporary directory as cp seems to support the <(cat files.txt) format, and from there grepping the files. Shirley there is a better way.

Solution 1

You seem to be grepping the list of filenames, not the files themselves. <(cat files.txt) just lists the files. Try <(cat $(cat files.txt)) to actually concatenate them and search them as a single stream, or

grep -i 'foo' $(cat files.txt)

to give grep all the files.

However, if there are too many files on the list, you may have problems with number of arguments. In that case I'd just write

while read filename; do grep -Hi 'foo' "$filename"; done < files.txt

Solution 2

xargs grep -i -- foo /dev/null < files.txt

assuming files are blank or newline delimited (where quotes or backslashes can be used to escape those separators). With GNU xargs you can specify the delimiter with -d (which then disables the quoting handling though).

(unset -v IFS; set -f; grep -i -- foo $(cat files.txt))

assuming files are space, tab or newline separated (no way to escape those though you can choose a different separator by assigning it to IFS). That one will fail if the file list is too big on most systems.

Those also assume that none of the files are called -.

Solution 3

To read a list of file names from stdin you can use xargs. E.g.,

cat files.txt | xargs -d'\n' grep -i -- 'foo'

By default, xargs reads items from the standard input, delimited by blanks. The -d'\n' tells it to use newline as the argument delimiter, so it can handle file names containing blanks. (As Stéphane Chazelas points out, that's a GNU extension). However, it won't cope with file names containing newlines; we'd need a slightly more complicated approach to handle those.

FWIW, this approach is somewhat faster than a while read loop, as bash's read command is very slow - it reads its data character by character, whereas xargs reads its input more efficiently. Also, xargs only invokes the grep command as many times as it needs to, with each invocation receiving multiple file names, and that's more efficient than invoking grep individually for each file name.

See the xargs man page and the xargs info page for further details.

Solution 4

xargs can read items from a file (like your files.txt list) with it's option:

   --arg-file=file
   -a file
          Read items from file instead of standard input.  If you use this
          option, stdin remains unchanged when commands are  run.   Other‐
          wise, stdin is redirected from /dev/null.

So this should work too:

xargs -a files.txt grep -i 'foo'

or for spaces in filenames

xargs -d'\n' -a files.txt grep -i 'foo'
xargs -I{} -a files.txt grep -i 'foo' {}

View more solutions

Comments

[dotancohen]

Thank you! I did not realize that while could receive the lines of file.txt as such.

[Stéphane Chazelas]

You'll want to disable the glob part of that split+glob operator here (unless the shell is zsh).

[Stéphane Chazelas]

To read a (text) line, the syntax is IFS= read -r filename, read filename is something else.

[Stéphane Chazelas]

Note that -H is a GNU extension. You're missing some --.

[PM 2Ring]

@dotancohen: For more info on read -r see unix.stackexchange.com/q/18886/88378 and unix.stackexchange.com/q/87301/88378

[jimmij]

It is better/faster to use $(< file) instead of $(cat file), at least in bash and zsh.