pipe stdout and stderr to two different processes in shell script?

Question

I've a pipline doing just

 command1 | command2

So, stdout of command1 goes to command2 , while stderr of command1 go to the terminal (or wherever stdout of the shell is).

How can I pipe stderr of command1 to a third process (command3) while stdout is still going to command2 ?

Solution 1

Use another file descriptor

{ command1 2>&3 | command2; } 3>&1 1>&2 | command3

You can use up to 7 other file descriptors: from 3 to 9.
If you want more explanation, please ask, I can explain ;-)

Test

{ { echo a; echo >&2 b; } 2>&3 | sed >&2 's/$/1/'; } 3>&1 1>&2 | sed 's/$/2/'

output:

b2
a1

Example

Produce two log files:
1. stderr only
2. stderr and stdout

{ { { command 2>&1 1>&3; } | tee err-only.log; } 3>&1; } > err-and-stdout.log

If command is echo "stdout"; echo "stderr" >&2 then we can test it like that:

$ { { { echo out>&3;echo err>&1;}| tee err-only.log;} 3>&1;} > err-and-stdout.log
$ head err-only.log err-and-stdout.log
==> err-only.log <==
err

==> err-and-stdout.log <==
out
err

Solution 2

The accepted answer results in the reversing of stdout and stderr. Here's a method that preserves them (since Googling on that purpose brings up this post):

{ command 2>&1 1>&3 3>&- | stderr_command; } 3>&1 1>&2 | stdout_command

Notice:

• 3>&- is required to prevent fd 3 from being inherited by command. (As this can lead to unexpected results depending on what command does inside.)

Parts explained:

1. Outer part first:

   1. 3>&1 -- fd 3 for { ... } is set to what fd 1 was (i.e. stdout)
   2. 1>&2 -- fd 1 for { ... } is set to what fd 2 was (i.e. stderr)
   3. | stdout_command -- fd 1 (was stdout) is piped through stdout_command

2. Inner part inherits file descriptors from the outer part:

   1. 2>&1 -- fd 2 for command is set to what fd 1 was (i.e. stderr as per outer part)
   2. 1>&3 -- fd 1 for command is set to what fd 3 was (i.e. stdout as per outer part)
   3. 3>&- -- fd 3 for command is set to nothing (i.e. closed)
   4. | stderr_command -- fd 1 (was stderr) is piped through stderr_command

Example:

foo() {
    echo a
    echo b >&2
    echo c
    echo d >&2
}

{ foo 2>&1 1>&3 3>&- | sed -u 's/^/err: /'; } 3>&1 1>&2 | sed -u 's/^/out: /'

Output:

out: a
err: b
err: d
out: c

(Order of a -> c and b -> d will always be indeterminate because there's no form of synchronization between stderr_command and stdout_command.)

Solution 3

Using process substitution:

command1 > >(command2) 2> >(command3)

See http://tldp.org/LDP/abs/html/process-sub.html for more info.

Solution 4

Simply redirect stderr to stdout

{ command1 | command2; } 2>&1 | command3

Caution: commnd3 will also read command2 stdout (if any).
To avoid that, you can discard commnd2 stdout:

{ command1 | command2 >/dev/null; } 2>&1 | command3

However, to keep command2 stdout (e.g. in the terminal),
then please refer to my other answer more complex.

Test

{ { echo -e "a\nb\nc" >&2; echo "----"; } | sed 's/$/1/'; } 2>&1 | sed 's/$/2/'

output:

a2
b2
c2
----12

Solution 5

Pipe stdout as usual, but use Bash process substitution for the stderr redirection:

some_command 2> >(command of stderr) | command of stdout

Header: #!/bin/bash

View more solutions

Comments

[FatalError]

Whoops, good call. I intially thought the OP wanted stderr to go only to command 3. This looks like the right way to go.

[FatalError]

Looks like the OP wanted both stdout and stderr to go to command2, which I initially missed. The above separates the two and send each separately to a command. I'll leave it though, as it might be useful to somebody.

[Admin]

No, I do not want both stdout and stderr to go to command2. stdout of command1 to command2, stderr of command1 to command3. command2 should not get stderr of command1

[Admin]

Wouldn't { command1 | command2 >/dev/null 2>&1 } 2>&1 | command3 prevent stdout/stderr of command2 to reach command3 , or would that also mess with stderr of command1 ?

[oHo]

Hi @user964970. The /dev/null redirection is a good idea. As you said, your example above mess stderr and stdout because they are inverted in the same step. I would prefer { command1 | command2 >/dev/null; } 2>&1 | command3. I edit my answer to use your brilliant contribution. Thanks ;-)

[Isaac Betesh]

How do you add a file descriptor? echo out >&3 outputs "-bash: 3: Bad file descriptor"

[Isaac Betesh]

Found the answer here: unix.stackexchange.com/questions/18899/…

[Rahul Kadukar]

This thing works, I verified it but I am not able to understand how it works. In the outer part, Point 3 stdout_command isn't fd1 now pointing to stderr, how is stdout going there instead of stderr.

[Rahul Kadukar]

Infact this also worked (command 2>&1 | stderr_command; ) 1>&2 | stdout_command

[jxy]

antak's answer below is more complete. It still maintains the original separation between stdout and stderr as the command does without all the pipes. Note that with pipes, command is run in a subprocess. If you don't want that, for you may want the command to modify global variables, you would need to create fifo and use redirections instead.

[josch]

Note: this is not POSIX but a bashism.

[antak]

@RahulKadukar That puts both stdout and stderr of command through stderr_command and nothing goes through stdout_command.

[jwd]

I enjoyed unraveling this, thank you (: Note: you could make it a little shorter by having the innermost redirects be merely 2>&3 3>&-. This does, however, mean you need to handle stdout on the inside of the curlies and stderr on the outside (so, swap stdin_command and stdout_command in your example).

[antak]

@jwd Thanks for the comment. :) Problem with that approach is stdout and stderr of the entire command line comes out reversed. I tested it by adding >/dev/null on the end of the command line and seeing if only a and c were filtered out.

[jwd]

Oh good point. And I guess there's no easy way to get around that since the outermost pipe will write to the shell's stdout regardless of redirects (unless we added more curlies). No golf points for me (:

[jbyler]

This is great, thanks @antak! But I found that while it works great in bash, it doesn't work as-is in zsh due to multios. I posted a tweaked version for zsh as another answer.

[Mark G.]

My quick’n’dirty go-to for stuff like this (particularly when I’m just typing — and thinking — left-to-right at an interactive shell) has usually been something along the lines of { cmd 3>&1 1>&2 2>&3 3>&- | tee err.log; } 3>&1 1>&2 2>&3 3>&- | tee out.log (swap out/err, process err; swap again, process out…) but the succinctity of this is quite appealing, and I suspect it’s more efficient to boot, since it allows the subshell to inherit the temporary file descriptor (as subprocesses were designed to do) instead of taking pains to recreate it unnecessarily… I may have to start using this. :)

[DeusXMachina]

This worked in zsh for me, except stderr and stdout commands were flipped for me. Both bash and zsh. It looks like to me you are redirecting stdout to 3, then routing 3 back to stdout, so the last statement should be to stdout. I have no idea.

[Kevin Keane]

One problem with this answer is that the { } creates a subshell, which in some situations is not acceptabe. For instance, you cannot pass variables back out of the {}.

[Andrevinsky]

Thanks, @oHo. BTW, is there a way to preserve the command's exit code, esp. given the tee eats it up. I.e. a next command rc=$? saves 0 to rc.

[goji]

It's also so much nicer than the POSIX solutions.