Why does this not work? "ls *.txt | xargs cat > all.txt" (all files into single txt document)

pipe cat xargs

41,129

Solution 1

ls *.txt | xargs cat >> all.txt

might work a bit better, since it would append to all.txt instead of creating it again after each file.

By the way, cat *.txt >all.txt would also work. :-)

Solution 2

If some of your file names contain ', " or space xargs will fail because of the separator problem

In general never run xargs without -0 as it will come back and bite you some day.

Consider using GNU Parallel instead:

ls *.txt | parallel cat > tmp/all.txt

or if you prefer:

ls *.txt | parallel cat >> tmp/all.txt

Learn more about GNU Parallel http://www.youtube.com/watch?v=OpaiGYxkSuQ

Solution 3

all.txt is a file in the same directory, so cat gets confused when it wants to write from the same file to the same file.

On the other hand:

ls *.txt | xargs cat > tmp/all.txt

This will read from textfiles in your current directory into the all.txt in a subdirectory (not included with *.txt).

41,129

Author by

ajo

Updated on September 17, 2022

Comments

ajo almost 2 years
Why does this not work?
```
ls *.txt | xargs cat > all.txt
```
(I want to join the contents of all text files into a single 'all.txt' file.) find with -exec should also work, but I would really like to understand the xargs syntax.

Thanks
- tripleee almost 5 years
  
  Though don't use ls for this. If you really can't use cat *.txt >all.txt then try printf '%s\0' *.txt | xargs -r0 cat >all and then mv all all.txt to avoid having the file referencing itself.
ajo almost 14 years

The cat *.txt >all.txt is naturally better. Thanks
ajo almost 14 years

However, the ... | xargs cat >> all.txt or > all.txt always return error with xargs: unmatched single quote ... Is it because xargs takes everything after it as the command?
Janne Pikkarainen almost 14 years

Do you have filenames with spaces? If so, then use something like "find /your/path -iname '*.txt' -print0 | xargs -0 cat >>all.txt" instead
ajo almost 14 years

Still the following error: xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option
ajo almost 14 years

no, I replaced all the filename spaces with . But thinking of it, some filenames are likely to include single quotes as in listing_O'Connor.txt, this might be the problem!
Janne Pikkarainen almost 14 years

Yes, that's the problem then. :) The easiest and the sanest way is to use find with -print0 combined with xargs -0 -- then the whole chain will use NULL character as a separator and whitespace and special characters will be taken care of automatically.
ajo almost 14 years

Indeed: After removing the single-quotes in some filenames via "s/'/_/g" *.txt, the command works OK!! But could it be done from within xargs via some option???
ajo almost 14 years

OK, find isn't bad either... I remember having similar problems when the file names contained spaces!!
Janne Pikkarainen almost 14 years

find is much better than ls in case you need to recurse into subdirectories.
ajo almost 14 years

for the latter: what about ls -R? (apart from the directory line?!)
Janne Pikkarainen almost 14 years

ls -R maybe fine for human readable form, but if you need to handle something with xargs or other tools -- not so much. See, ls -R does not list the full path along with the every filename, but find or tree will do it. Makes scripting a lot easier. When scripting or piping stuff, please get rid of ls and use more advanced tools :-)
dwlz almost 10 years

This is potentially very dangerous command. If "all.txt" already exists, running this command will expand to fill all available hard drive space.