cat: write error: Broken pipe

Question

I'm running a command:

cat urls.txt | xargs -L 1 -P 5 timeout 40 wget

This downloads all the URLs in urls.txt

This used to work fine, but recently I changed the URLs I am downloading from something like: domain.com/54f-5g4/ to something like domain.com/this-that/

And for some reason I am now getting:

cat: write error: Broken pipe

I've looked at the commands being executed and they all look fine, i.e. no unescaped characters.

Does anyone have any idea what the problem may be? Or how to get more debugging information? I believe the problem is with commands after the pipe.

Update: I found the error. It looks like I accidently typed a ' in my urls.txt file. If you are getting this error, you have a problem with your command after the pipe.

Solution

When you use a pipe in a shell:

command_1 | command_2

the output of command_1 is piped to command_2 as input. If command_2 ends before reading all the output of command_1 (e.g. in your case because it found something wrong as you said in your update that you accidentally typed a '), then command_1 may complain that its output pipe was prematurely closed.

Comments

[AzkerM]

Correct me if I'm wrong.. If your purpose is to download whatever specified on that urls.txt file; well you can simply use wget -i urls.txt :)

[Kohjah Breese]

wget only allows one connection at a time, using xargs, you can have multiple connections running simultaneously.

[Krzysztof Krasoń]

Just a small follow up, why this happens on some systems and doesn't on others. I mean the message e.g. is not shown on my bash, but is shown on my server.