How can I download multiple files stored in a text file with curl and xargs?
11,613
Solution 1
I found solution:
cat ./../c | xargs -n1 curl -O
xargs splits stdin by spaces and newlines, and passes to curl at once. So curl called only once with long arguments.
n1
option limits this passing argument count as 1, so curl will be called multiple times.
Solution 2
Using GNU Parallel http://www.gnu.org/software/parallel/ you can do:
cat listfile.txt | parallel curl -O
Not only does GNU Parallel deal nicely with special chars like ' " and space, you will also get the added benefit of downloading in parallel.
Watch the intro video to GNU Parallel: http://www.youtube.com/watch?v=OpaiGYxkSuQ
Related videos on Youtube
Author by
Eonil
Updated on September 17, 2022Comments
-
Eonil almost 2 years
How can I download multiple files stored in a text file with curl and xargs? This is my last trial:
cat listfile.txt | xargs curl -O
first file works well, but other files are just output to stdout.
-
Eonil almost 14 yearsNice suggestion. But I have no time ti test this, so +1, and thanks!
-
Mr.Gando about 11 yearsThis is awesome.