How to grep a program's output but also echo the output normally?

Question

I'm working with a program that outputs error messages when something goes wrong, but does not set its exit status accordingly: the exit status is always 0, indicating success. I'd like to run this program from a shell script and get a non-zero exit status if it emits any error messages, so the script can tell that the program failed.

The error messages follow a predictable pattern that I can match with grep, and grep sets its exit status based on whether it found a match, so I can pipe the program's output into grep and negate the result with ! to get the exit status I want.

The problem is that if I pipe into grep, I can't see the program's output anymore, because grep consumes it. I'd like to somehow scan the output for error messages but also display the output normally, since there are other important messages besides the errors. Unfortunately, grep doesn't have an option to pass-through all of its input, matching and non-matching lines alike.

One approach I've found that mostly works is:

! my_program | tee /dev/stderr | grep -q "error message"

This feeds the program's output into grep but also copies it to standard error, which isn't redirected, so I can see it. That's OK when my script's standard output and standard error both go to a terminal (so it doesn't matter which one gets the messages), but it can cause problems if standard out and standard error are redirected to different places; the messages will end up in the wrong place.

Using bash or POSIX shell, and GNU tools, is there a (concise) way to scan a program's standard out and/or standard error streams with something like grep and set exit status accordingly, but also let both streams go to their normal destinations?

(Note that my_program writes its error messages to standard out, not standard error, so a solution that can only scan the standard output stream is OK, though not ideal. And in case it makes a difference, I'm doing this on CentOS 7.)

Solution 1

...is there a (concise) way to scan a program's standard out and/or standard error streams with something like grep and set exit status accordingly, but also let both streams go to their normal destinations?

...my_program writes its error messages to standard out, not standard error, so a solution that can only scan the standard output stream is OK, though not ideal.

My solution answers the bolded portion above.

I think the easiest way to do this is through awk:

myprogram |
awk 'BEGIN {status = 0} /error message/ {status = 1} 1; END {exit(status)}'

The awk command outputs everything it inputs exactly as-is, but at the end it exits with a status dependent on whether "error message" was a part of the input.

Concise version (with a shorter variable name):

myprogram | awk 'BEGIN{s=0} /error message/{s=1} 1; END{exit(s)}'

Solution 2

Here are some one liners that grep the error message $MSG on stdout/stderr, and do/do not stop the program as soon as the error message appears:

# grep on stdout, do not stop early
my_program | awk -v s="$MSG" '$0~s{r=1} 1; END{exit(r)}'

# grep on stdout, do stop early
my_program | awk -v s="$MSG" '$0~s{exit(1)} 1'

# grep on stderr, do not stop early
{ my_program 2>&1 >&3 | awk -v s="$MSG" '$0~s{r=1} 1; END{exit(r)}' >&2; } 3>&1

# grep on stderr, do stop early
{ my_program 2>&1 >&3 | awk -v s="$MSG" '$0~s{exit(1)} 1' >&2; } 3>&1

Notes:

• For all: the stream on which you apply grep will lose its potential tty status, because my_program sees it going into awk. This might affect how my_program writes to this stream, e.g., it might not print rotating progress bars because it might assume it cannot control cursor position.

• For all: stdout and stderr will not be merged, and can be redirected independently from each other, as usual.

• For all: the original exit code of my_program gets completely ignored. The only other easy alternative is: exit with error if either my_program exits with error, or the error message appears. To get this behaviour, you need to enable pipefail in the Bash shell that executes the pipeline. E.g.:

  (set -o pipefail; my_program | awk -v s="$MSG" '$0~s{exit(1)} 1')
  

• For grepping on stderr: the simple command lines above assume file descriptor 3 is unused. That's usually an ok assumption to make. To avoid this assumption, you can get Bash to allocate a file descriptor that's guaranteed to be unused with:

  bash -c 'exec {fd}>&1; { my_program 2>&1 >&${fd} | awk -v s="$MSG" '\''$0~s{exit(1)} 1'\'' >&2; } ${fd}>&1'
  

Comments

[Julie Pelletier]

Why use 2 terminals? Maybe I misread the question but I didn't see anything related mentioned.

[John Aspinall]

To separate the two kinds of output: important stuff being matched by grep, vs. everything.

[Wyzard]

I need the output to actually come from the script that's running the program, like it would if the script just ran the program normally. (It's actually a Jenkins build script; Jenkins captures the script's output and saves it in a log. The exit status is so that Jenkins can tell whether the build has failed.)

[John Aspinall]

Ahh, I see. Then something like my next answer.

[agc]

The redirect juggling needs more explanations, to be added later, time permitting...

[Wildcard]

Correction to earlier comment: It is possible, but not with standard utilities. So, not fundamentally impossible. (To do so, you could use mkfifo to set up the stdout and stderr targets, then open them for reading using a custom-written program which will read from both sources and write each to the appropriate file descriptor, like a dual-action cat, and check for a pattern in each along the way.)