Executing lines of a file with xargs, using stdout redirection

Question

I have a file that has lines that look like this:

"http://api.domain.com/path/to/resource?format=json" > output.xml

and I execute the following from my command line:

cat file | xargs -L 1 -I {} wget {}

Unhappily, the command looks lik it executes, but no file ever gets created. When I use the command:

cat file | xargs -L 1 -I {} echo {}

It prints the lines of the file without the double quotes. I think this may be the source of the problem. How do I get it to include them?

How can correctly I execute the wget command?

Solution

Quotes, redirection operators, etc. are typically parsed by your shell before the argv[] array is constructed. > is stripped away; the shell performs redirection. Quotes are stripped away; the shell performs word splitting.

With xargs, the entire input line goes directly to argv[], with the quotes and the operators being passed to wget, which has no idea what to do with them.

To parse such input files as you have, you'll need to get help from the shell, in two possible ways.

---

< file xargs -d '\n' -I {} sh -x -c "wget {}"

This way, file is being read line by line without allowing xargs to do any splitting (-d '\n') and passed to the sh shell as part of a command line. The -x option to sh lets you see commands it executes.

---

while read -r line; do
    eval "wget $line"
done < file

Here the same is done using only builtins of the current shell – either sh, bash or compatible.

Comments

[slhck]

In your case, why not prepend each line with wget and execute it?

[Zoredache]

Are you sure wget is the tool you are looking for here? Did you mean to include the -O option to specify the output file? Or maybe you want curl?