How can I pipe stderr, and not stdout?

Question

I have a program that writes information to stdout and stderr, and I need to process the stderr with grep, leaving stdout aside.

Using a temporary file, one could do it in two steps:

command > /dev/null 2> temp.file
grep 'something' temp.file

But how can this be achieved without temp files, using one command and pipes?

Solution 1

First redirect stderr to stdout — the pipe; then redirect stdout to /dev/null (without changing where stderr is going):

command 2>&1 >/dev/null | grep 'something'

For the details of I/O redirection in all its variety, see the chapter on Redirections in the Bash reference manual.

Note that the sequence of I/O redirections is interpreted left-to-right, but pipes are set up before the I/O redirections are interpreted. File descriptors such as 1 and 2 are references to open file descriptions. The operation 2>&1 makes file descriptor 2 aka stderr refer to the same open file description as file descriptor 1 aka stdout is currently referring to (see dup2() and open()). The operation >/dev/null then changes file descriptor 1 so that it refers to an open file description for /dev/null, but that doesn't change the fact that file descriptor 2 refers to the open file description which file descriptor 1 was originally pointing to — namely, the pipe.

Solution 2

Or to swap the output from standard error and standard output over, use:

command 3>&1 1>&2 2>&3

This creates a new file descriptor (3) and assigns it to the same place as 1 (standard output), then assigns fd 1 (standard output) to the same place as fd 2 (standard error) and finally assigns fd 2 (standard error) to the same place as fd 3 (standard output).

Standard error is now available as standard output and the old standard output is preserved in standard error. This may be overkill, but it hopefully gives more details on Bash file descriptors (there are nine available to each process).

Solution 3

In Bash, you can also redirect to a subshell using process substitution:

command > >(stdlog pipe)  2> >(stderr pipe)

For the case at hand:

command 2> >(grep 'something') >/dev/null

Solution 4

Combining the best of these answers, if you do:

command 2> >(grep -v something 1>&2)

...then all stdout is preserved as stdout and all stderr is preserved as stderr, but you won't see any lines in stderr containing the string "something".

This has the unique advantage of not reversing or discarding stdout and stderr, nor smushing them together, nor using any temporary files.

Solution 5

It's much easier to visualize things if you think about what's really going on with "redirects" and "pipes." Redirects and pipes in bash do one thing: modify where the process file descriptors 0, 1, and 2 point to (see /proc/[pid]/fd/*).

When a pipe or "|" operator is present on the command line, the first thing to happen is that bash creates a fifo and points the left side command's FD 1 to this fifo, and points the right side command's FD 0 to the same fifo.

Next, the redirect operators for each side are evaluated from left to right, and the current settings are used whenever duplication of the descriptor occurs. This is important because since the pipe was set up first, the FD1 (left side) and FD0 (right side) are already changed from what they might normally have been, and any duplication of these will reflect that fact.

Therefore, when you type something like the following:

command 2>&1 >/dev/null | grep 'something'

Here is what happens, in order:

1. a pipe (fifo) is created. "command FD1" is pointed to this pipe. "grep FD0" also is pointed to this pipe
2. "command FD2" is pointed to where "command FD1" currently points (the pipe)
3. "command FD1" is pointed to /dev/null

So, all output that "command" writes to its FD 2 (stderr) makes its way to the pipe and is read by "grep" on the other side. All output that "command" writes to its FD 1 (stdout) makes its way to /dev/null.

If instead, you run the following:

command >/dev/null 2>&1 | grep 'something'

Here's what happens:

1. a pipe is created and "command FD 1" and "grep FD 0" are pointed to it
2. "command FD 1" is pointed to /dev/null
3. "command FD 2" is pointed to where FD 1 currently points (/dev/null)

So, all stdout and stderr from "command" go to /dev/null. Nothing goes to the pipe, and thus "grep" will close out without displaying anything on the screen.

Also note that redirects (file descriptors) can be read-only (<), write-only (>), or read-write (<>).

A final note. Whether a program writes something to FD1 or FD2, is entirely up to the programmer. Good programming practice dictates that error messages should go to FD 2 and normal output to FD 1, but you will often find sloppy programming that mixes the two or otherwise ignores the convention.

View more solutions