Best way to follow a log and execute a command when some text appears in the log

Question

I have set the below parameters in sqlplus to display my view out put on unix box,However it is displaying one line gap between two records.I don't want that one line gap between two records Ex-

set feedback off
SET NEWPAGE NONE
set HEADING Off
set pagesize 0
set linesize 125
SET TRIMSPOOL ON
set termout off
spool /export/home/43276826/Rep_Tran_oracle_$DATE_FILE.txt
select RecordID||','||C_S||','||P_R||','||AccountingDate||','||SettlementDate||','||Sec
Description||','||ISIN||','||MessageRef||','||Amount||','||Department||','||AssignedTo||','||LastUpdate||','||CashAmount||','
||CashAmountUSD||','||LastNoteText||','||LastNoteUser||','||CashCurrency||','||BIMASNumber||','||RelatedReference||','||Sende
rToRec||','||OpType||','||OriginalISIN from HSBC_ALL_OI_T;

201280,C,R,21.4.2009,21.4.2009,"HSBC HLDG","GB0005405286","00001/20090421-1006851",188.00
0000,"TITBB/F"," ",22.4.2009,0.00,,"NOTHING GENEVA","ADK"," ","GB411161","SF-0357690","  ","FR"," "

"201279,C,P,21.4.2009,21.4.2009,"HSBC HLDG","GB0005405286","00001/20090421-1401548",188.00
0000,"TITBB/F"," ",22.4.2009,0.00,,"NOTHING GENEVA","ADK"," ","GB411161","SF-0357689","  ","FD"," "

there is a gap of one line between two records,I don't want that one line gap.I want output in the below format:

201280,C,R,21.4.2009,21.4.2009,"HSBC HLDG","GB0005405286","00001/20090421-1006851",188.00
0000,"TITBB/F"," ",22.4.2009,0.00,,"NOTHING GENEVA","ADK"," ","GB411161","SF-0357690","  ","FR"," "
"201279,C,P,21.4.2009,21.4.2009,"HSBC HLDG","GB0005405286","00001/20090421-1401548",188.00
0000,"TITBB/F"," ",22.4.2009,0.00,,"NOTHING GENEVA","ADK"," ","GB411161","SF-0357689","  ","FD"," "

Please help me out in achieving the above output.

Solution 1

A simple way would be awk.

tail -f /path/to/serverLog | awk '
                    /Printer is on fire!/ { system("shutdown -h now") }
                    /new USB high speed/  { system("echo \"New USB\" | mail admin") }'

And yes, both of those are real messages from a kernel log. Perl might be a little more elegant to use for this and can also replace the need for tail. If using perl, it will look something like this:

open(my $fd, "<", "/path/to/serverLog") or die "Can't open log";
while(1) {
    if(eof $fd) {
        sleep 1;
        $fd->clearerr;
        next;
    }
    my $line = <$fd>;
    chomp($line);
    if($line =~ /Printer is on fire!/) {
        system("shutdown -h now");
    } elsif($line =~ /new USB high speed/) {
        system("echo \"New USB\" | mail admin");
    }
}

Solution 2

If you're only looking for one possibility and want to stay mostly in the shell rather than using awk or perl, you could do something like:

tail -F /path/to/serverLog | 
grep --line-buffered 'server is up' | 
while read ; do my_command ; done

...which will run my_command every time "server is up" appears in the log file. For multiple possibilities, you could maybe drop the grep and instead use a case within the while.

The capital -F tells tail to watch for the log file to be rotated; i.e. if the current file gets renamed and another file with the same name takes its place, tail will switch over to the new file.

The --line-buffered option tells grep to flush its buffer after every line; otherwise, my_command may not be reached in a timely fashion (assuming the logs have reasonably sized lines).

Solution 3

It is strange that no one mentioned about multitail utility which has this functionality out-of-box. One of usage example:

Show the output of a ping-command and if it displays a timeout, send a message to all users currently logged in

multitail -ex timeout "echo timeout | wall" -l "ping 192.168.0.1"

See also another examples of multitail usage.

Solution 4

This question appears to be answered already, but I think there's a better solution.

Rather than tail | whatever, I think what you really want is swatch. Swatch is a program designed explicitly for doing what you're asking, watching a log file and executing actions based on log lines. Using tail|foo will require that you've got a terminal actively running to do this. Swatch on the other hand runs as a daemon and will always be watching your logs. Swatch is available in all Linux distros,

I encourage you to try it out. While you can pound a nail in with the back side of a screwdriver does not mean you should.

The best 30-second tutorial on swatch I could find is here.

Solution 5

bash could do the job by himself

Let see how simple and readable it could be:

mylog() {
    echo >>/path/to/myscriptLog "$@"
}

while read line;do
    case "$line" in
        *"Printer on fire"* )
            mylog Halting immediately
            shutdown -h now
            ;;
        *DHCPREQUEST* )
            [[ "$line" =~ DHCPREQUEST\ for\ ([^\ ]*)\  ]]
            mylog Incomming or refresh for ${BASH_REMATCH[1]}
            $HOME/SomethingWithNewClient ${BASH_REMATCH[1]}
            ;;
        * )
            mylog "untrapped entry: $line"
            ;;
    esac
  done < <(tail -f /path/to/logfile)

While you don't use bash's regex, this could stay very quick!

But bash + sed is a very efficient and interesting tandem

But for high load server, and as I like sed because it's very quick and very scalable, I often use this:

while read event target lost ; do
    case $event in
        NEW )
            ip2int $target intTarget
            ((count[intTarget]++))
        ...

    esac
done < <(tail -f /path/logfile | sed -une '
  s/^.*New incom.*from ip \([0-9.]\+\) .*$/NEW \1/p;
  s/^.*Auth.*ip \([0-9.]\+\) failed./FAIL \1/p;
  ...
')

View more solutions

Comments

[s g]

be sure to use tail -F to handle log rotation - i.e. my.log becomes full and moves to my.log.1 and your process creates a new my.log

[blong]

Here's a related discussion in the Ubuntu Q&A site: askubuntu.com/q/1052891/31592

[Admin]

thank you very much lotsoffreetime..Now it is giving me the right result set!!

[jonderry]

I like the awk solution for being short and easy to do on the fly in one line. However, if the command I want to run has quotes, this can be a bit cumbersome, is there an alternative, maybe using pipelines and compound commands that also allows a brief one-liner solution but doesn't require the resulting command to be passed as a string?

[penguin359]

@jon You can write awk as a script. Use "#!/usr/bin awk -f" as the first line of the script. This will eliminate the need for the outer single quotes in my example and free them for use inside a system() command.

[jonderry]

@penguin359, True, but it'd still be interesting to do it from the command line as well. In my case, there are a variety of different things I'd want to do including many things I can't foresee, so it's convenient to be able to just start the server and do it all in one line.

[jonderry]

I found an alternative, though I don't know how solid it is: tail -f /path/to/serverLog | grep "server is up" | head -1 && do_some_command

[penguin359]

@jon That seems a little fragile using head that way. More importantly, it's not repeatable like my examples. If "server is up" is in the last ten lines of the log, it will fire the command and exit immediately. If you restart it, it will most likely fire and exit again unless ten lines not containing "server is "up have been added to the log. A modification of that that might work better is tail -n 0 -f /path/to/serverLog That will read the last 0 lines of the file, then wait for more lines to print.

[penguin359]

However, if your goal is to fire a command every single time something appears in the log files, both version of tail and head may miss lines between each invocation.

[jonderry]

Thanks, -n 0 will help. In my current case, a little fragility isn't an issue because I'm just going through experimental iteration. From another question, I found that grep -m 1 "server is up" <(tail -n 0 -f /path/to/serverLog); some_command avoids problems with a pipe remaining alive when I expect it to die and have the command terminate.

[tcoolspy]

+1, I had no idea `multitail had those kind of ninja skills tucked away. Thanks for pointing that out.

[penguin359]

@jon You may stil want && instead of ;. The && means execute the next command if and only if the previous command was successful. The ; means execute the second command when the first command finishes. For example, if I hit Ctrl-Z I get a funny behavior and the second command executes immediately. With && this doesn't happen.

[F. Hauri]

Syntaxe tail -f ... | perl -ne ... may present same simple syntax than using ... | awk .... The most important part of your script is tail-into-perl.

[s g]

Use tail -F (capital f) to account for rotating logs. i.e. my.log becomes full and moves to my.log.1 and your process creates a new my.log

[doctaphred]

I really like this answer, but it didn't work for me at first. I think you need to add the --line-buffered option to grep, or otherwise make sure it flushes its output between lines: otherwise, it just hangs, and my_command is never reached. If you prefer ack, it has a --flush flag; if you prefer ag, try wrapping with stdbuf. stackoverflow.com/questions/28982518/…

[Sun]

WebArchive is your friend: web.archive.org/web/20140922041805/http://campin.net/…

[Machtyn]

I tried this, using do exit ;. It seemed to work fine, but tail never finished and our script never moved onto the next line. Is there a way to stop tail in the do section?

[Eric]

You could instead add the option -n 0 to tail instead of having the sed

[Eric]

Also, it seems you need the --line-buffered option on grep to make it work

[Jeff Schaller]

Perhaps a small example would benefit the OP & others? The question starts with: tail -f /path/to/serverLog | grep "server is up" ... and then execute a command.

[Keith]

If you need the log line in my_command, you can use ... while read line; do my_command $line ; done