Combine multiple unix commands into one output

Question

I need to search our mail logs for a specific e-mail address. We keep a current file named maillog as well as a week's worth of .bz2 files in the same folder. Currently, I'm running the following commands to search for the file:

grep [email protected] maillog
bzgrep [email protected] *.bz2

Is there a way combine the grep and bzgrep commands into a single output? That way, I could pipe the combined results to a single e-mail or a single file.

Solution 1

Another way is

{ grep ...; bzgrep ...;} >file

&& has the difficulty that the bzgrep wouldn't be run if the grep failed.

Note the mandatory space after the opening curly brace and semicolon after the last command. Alternatively, you can use the subshell syntax (parentheses instead of curly braces), which isn't as picky:

(grep ...; bzgrep ...) >file

Solution 2

bzgrep automatically defaults to regular grep if a file isn't bzip-compressed. Thus the following should be sufficient:

bzgrep [email protected] maillog *bz2 | mail -s "logs yay" someuser@blah

oh also of course here's my obligatory GNU Parallel solution too:

parallel -m bzgrep [email protected] ::: maillog* *bz2 | mail -s "logs yay" someuser@blah

which could be a lot faster if you are checking a lot of files.

Solution 3

Here's another way to do it (assuming you're running bash, which you probably are):

cat <(bzgrep ...) <(grep ...)

Here bash is transparently feeding the output of the bzgrep and grep commands into cat as if they were files (and they sort of are under the hood, details in url at the bottom).

In your particular case I'd recommend Phil's solution, but the above is a good trick to keep in your bag.

If you're interested, you can read more here: http://www.tldp.org/LDP/abs/html/process-sub.html

Solution 4

At the time of my writing this, the accepted answer's syntax was wrong for most, if not all, Bourne-derived shells, including bash. I suggested an edit to the top and accepted answer to fix that, but I was also inclined to add all this other information, and this would've been more of a rewrite instead of an edit.

You can use compound commands:

{ grep ...; bzgrep ...; } >file

..or subshells (note the parentheses instead of curly braces):

(grep ...; bzgrep ...) >file

..to group the commands. The subshell way has nicer syntax (more forgiving of lack of whitespace and allows you to omit the last semicolon), but it either forks a new process, or "pretends" to by having the commands run in an cleaned up environment. Both have advantages depending on what you want to do, which don't matter here, but are worth looking up if you want more proficiency with the shell.

Note: You can use pipelining with these tricks too, so you could do something like this:

{ grep ...; bzgrep ...; } | less

P.S. if you don't care about the ordering of the matches in your combined output, you could use a single & between the two commands, like so: { grep ... & bzgrep ...; }. Then the two commands run simultaneously: the grep gets launched and the shell puts it in the background, then the shell will run the bzgrep. (But there's a small caveat with that, with the explanation involving file redirection and file stream buffering potentially causing a very small portion of the lines in the output file to get split/mangled: whether you'd see this would depend on how your grep, bzgrep, and libc stdio.h functions are implemented. In most implementations, I believe piping the command before redirecting to a file will avoid the problem, so you could do { foo & bar; } | cat - >file as a workaround.)

View more solutions

Comments

[DerfK]

As geekosaur mentioned, && shouldn't be used here because grep's return value depends on whether it found something or not. If you did grep ... && bzgrep ..., if grep had no hits, it would return failure and the command would stop. >> is a good idea though, unlike >, it will add output to the end of an existing file.

[Mike]

right I had forgotten about the exit condition preventing the second command.

[Phil Hollenback]

Ah yes, the canonical answer for 'how do you diff the output of two processes'. A good trick to know.

[Chris Magnuson]

On arch linux this worked with () but not {}, either way +1 as this is what I needed!