Writing "tail -f" output to another file
Solution 1
Buffering is the problem.
Do it this way,
tail -f log.txt | egrep --line-buffered 'WARN|ERROR' | tee filtered_output.txt # ^^^^^^^^^^^^^^^
Confirmed to work on Cygwin too.
Solution 2
It's probably a buffering issue. See this SO post on disabling the auto-buffering when using pipes. You can use the unbuffer
command from expect
:
$ unbuffer tail -f log.txt | egrep 'WARN|ERROR' | tee filtered_output.txt
Edit: Since you have a longer pipeline, you probably need to unbuffer each command (except the last):
$ unbuffer tail -f log.txt | unbuffer egrep 'WARN|ERROR' | tee filtered_output.txt
Edit 2: unbuffer
is available on Cygwin from the expect
source package (eg expect-20030128-1-src.tar.bz2, found in the expect/examples
folder), but it's a very short script. If you have the expect
package already installed, simply put this into a script called unbuffer
in your /usr/local/bin
directory:
#!/usr/bin/expect --
# Description: unbuffer stdout of a program
# Author: Don Libes, NIST
eval spawn -noecho $argv
set timeout -1
expect
On Debian, the unbuffer
command is provided in the expect-dev
package and is installed as expect_unbuffer
.
Solution 3
When using a command that does not really 'finish' (such as tail -f
), this actually does not really work or that well (at all).
You should be able to redirect the output to a text file. Try this:
tail -f log.txt | egrep 'WARN|ERROR' > filtered_output.txt
Solution 4
As others have pointed out, you can use the unbuffer
utility from Expect.
Note, however, that depending on your system and available version of Expect, you may need to use the -p
switch to unbuffer. Citing the man page:
Normally, unbuffer does not read from stdin. This simplifies use of unbuffer in some situations. To use unbuffer in a pipeline, use
the -p flag. Example:
process1 | unbuffer -p process2 | process3
So you might need this invocation:
unbuffer -p tail -f log.txt | unbuffer -p egrep 'WARN|ERROR' | tee filtered_output.txt
BTW, see this article for thorough explanation of the output buffering problem: http://www.pixelbeat.org/programming/stdio_buffering/
Solution 5
This is the version of unbuffer
that I have:
#!/usr/bin/expect --
# Description: unbuffer stdout of a program
# Author: Don Libes, NIST
if {[string compare [lindex $argv 0] "-p"] == 0} {
# pipeline
set stty_init "-echo"
eval spawn -noecho [lrange $argv 1 end]
close_on_eof -i $user_spawn_id 0
interact {
eof {
# flush remaining output from child
expect -timeout 1 -re .+
return
}
}
} else {
set stty_init "-opost"
set timeout -1
eval spawn -noecho $argv
expect
}
Related videos on Youtube
Mike
Updated on September 17, 2022Comments
-
Mike over 1 year
As a continuation from my last post where I have used
grep & tail -f
to find occurences of "rare" events. I would like to record this in another file.I've tried turning
tail -f log.txt | egrep 'WARN|ERROR'
into
tail -f log.txt | egrep 'WARN|ERROR' | tee filtered_output.txt
The file gets created, but nothing is populated, is this a caching issue or otherwise? How would I get a real-time appending of my tail's output to a new file?
-
Mike over 14 yearsThis doesn't seem to work.
-
Mike over 14 yearsis there a way to accomplish this with cygwin?
-
quack quixote over 14 yearsadded info on using in cygwin; you'll need the
expect
package. -
Mike over 14 yearsThanks, can't try until Monday now sadly. Will update then.
-
quack quixote over 14 years+1 thanks for the additional info. dunno if it's cygwin-friendly but it looks like a more intelligent script.