Tee doesn't get whole output from the pipe

Question

I have a script executing commands like:

export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH;./some_app -i $INDEX | tee $LOG
echo "Number of errors: $(grep "ERROR" $LOG | wc -l)"

The problem is probably in the pipe to tee. It does not seem to get the whole output. When the application quits last few lines of the output (usually those containing a fatal error) are missing. When I run the app without pipe to tee I get them in the output.

How can I force script to wait for tee to complete processing of all output?

Solution 1

The fatal error is probably coming out in STDERR (2), not STDOUT (1). You can redirect STDERR into STDOUT with 2>&1 and then the pipe should capture it too.

./some_app -i $INDEX 2>&1 | tee $LOG

If you have buffering issues on top, you could force it into an unbuffered state:

stdbuf -o0 ./some_app -i $INDEX 2>&1 | tee $LOG

Solution 2

As the error messages are normally shown on STDERR (File descriptor 2), you need to redirect both the STDOUT and STDERR to tee:

./some_app -i "$INDEX" |& tee "$LOG"

When you do ./some_app -i $INDEX | tee $LOG you are only redirecting the STDOUT to tee.

|& will cause both the STDOUT and STDERR to be redirected.

If you cant to redirect only the STDOUT (As you were):

./some_app -i "$INDEX" | tee "$LOG"

On the other hand if you want to redirect only the STDERR:

./some_app -i "$INDEX" 2>&1 >/dev/null | tee "$LOG"

Comments

[Admin]

Does it work OK if you tee it to a file, not stdout?

[Ladislav Mrnka]

Good, we are getting closer. Now I see that fatal error is being printed but again it is not complete. The line with the error just ends in the middle and echo output continues. There is still some problem with flushing buffer or simply waiting on that part to complete.

[Oli]

Edited. Fairly rare in my experience that something so completely slips through buffers on exit but it's worth a go.

[Ladislav Mrnka]

Done! Thank you. I may be asking too many question but does anyone understand why I need to turn off buffering when piping to other process?

[Pilot6]

@Oli A very good one!