how to split input to two pipes

Question

I would like to do something equivalent to this:

some-expensive-command > /tmp/mytempfile
grep -v "pattern" /tmp/mytempfile >> output.txt
grep "pattern" /tmp/mytempfile | yet-another-command

Preferably elegant and without the need for the tempfile. I was thinking about piping through tee, but the best I can think of might is to combine two of the three lines and still require the intermediate storage:

some-expensive-command | tee /tmp/mytempfile | grep -v "pattern" >> output.txt
grep "pattern" /tmp/mytempfile | yet-another-command

Solution 1

The way question reads it sounds like you want one stdin redirected to two different commands. If that's the case, take advantage of tee plus process substitution:

some-expensive-command | tee >(grep 'pattern' > output.txt) >(grep -v 'pattern' | another-command)

Process substitutions are in fact anonymous pipelines implemented within bash itself ( on subprocess level ). We can also make the use of a named pipeline + tee. For instance, in terminal A do

$ mkfifo named.fifo
$ cat /etc/passwd | tee named.fifo | grep 'root'

And in another terminal B do

$ grep -v 'root' named.fifo

---

Another way to look at this is by recognizing that grep is line pattern matching tool, so by reading line at a time and using that same line in multiple commands we can achieve exactly the same effect:

rm output.txt # get rid of file so that we don't add old and new output
some-expensive-command | while IFS= read -r line || [ -n "$line" ]; do
    printf "%s\n" "$line" | grep 'pattern' >> output.txt
    printf "%s\n" "$line" | grep -v 'pattern' | another-command
done
# or if another-command needs all of the output, 
# place `| another-comand` after `done` clause

---

Yet another way is to abandon grep and use something more powerful, like awk:

some-expensive-command | awk '/pattern/{print >> "output.txt"}; !/pattern/{print}' | another-command.

Practically speaking, don't worry about using temporary files, so long as you clean them up after using. If it works, it works.

Solution 2

Use bash Process Substitution:

some-command | tee >(grep "pat" | another-command >>out1) | grep -v "pat" >>out2

The process substitution assigns some-command’s output to grep "pat"’s input, thus saving you the tempfile. Of course the data is still saved in a file (it’s always), just that you don’t have to take care of that. If you don’t want to save another-command’s output in a file but rather print it I recommend to simply switch the two command lists.

Another nice source of information: man bash/EXPANSION

Comments

[Sergiy Kolodyazhnyy]

So you want one command's output saved to file output.txt and that same output redirected to another command for further processing ? Is that what you're trying to do ?