How does curl print to terminal while piping

Question

When I curl a file, and pipe it to a file or another command, I see output in my terminal. I am not sure how this happens, as the pipe is supposed to take all the output from curl, right?

For example:

$ curl http://www.archive.org/stream/Pi_to_100000000_places/pi.txt > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  129M    0  129M    0     0  22.5M      0 --:--:--  0:00:05 --:--:-- 24.7M

---

Edit

I use curl like this:

curl http://www.archive.org/stream/Pi_to_100000000_places/pi.txt | some_other_command > some_file

I do not want to pipe the status to some_other_command, I was just wondering how it was able to display the status. However, showing how to redirect both streams added to the answer, so don't remove that.

Solution 1

There are two output streams generally available: standard output, and standard error. In practice, when running in a terminal, both send data to the terminal. > only redirects standard output, and curl prints the progress data to standard error. To suppress both, use one of:

curl ... > /dev/null 2>&1
curl ... &> /dev/null    # bash's combined redirection operator
curl -s ...    # -s, --silent: Silent or quiet mode. Don't show progress meter or error messages.

To send both to a pipe:

curl ... 2>&1 | ...
curl |& ...    # bash's combined pipe

Unless you use the |& or &> operators, all streams are redirected independently.

Also see:

• Stack Overflow: confused about stdin, stdout and stderr?

Solution 2

When you are using curl to open an URL, you'll get two output:

1. The status of the curl itself.
2. The contents of that URL.

Curl should use a way to show these two separately otherwise processing of the real output (URL's content) would be hard and I'll end up with unnecessary contents (curl's status).

So it uses stderr for its status and stdout for the content.

Using > you are redirecting the URL's content (stdout) to the /dev/null, you should actually use: 2> /dev/null instead.

Also if you want to pip both of them to the next command:

curl url |&  command

If you only want the content be piped to next command while not seeing the status:

curl 2> /dev/null | command

Comments

[edwinksl]

See superuser.com/q/173209/53494.

[Eliah Kagan]

You're redirecting curl's output to /dev/null for testing, right? I'm asking because the answers so far all feature how to redirect its standard output and standard error streams together. But if you're using curl to download files on standard output, you do not want to redirect stdout and stderr to the same place--any output on stderr would corrupt the download! I know you've already accepted an answer, but would you be willing to edit this with an example of how you're using curl? I think that would make this question more valuable to future readers in similar situations.

[Alex]

@EliahKagan Sure, thanks for the suggestion. I am piping it to /dev/null just for demonstration

[Dino Douglas]

Maybe you want to do this? echo curl -s http://api.ipify.org | some_command

[Daniel Black]

Seems like this also works with &>> to output to a file. As mentioned without putting curl into silent mode then the contents include the progress data, but I actually want that in my case.