In tail -f, how do I filter out stuff that has certain keywords?

35,598

Solution 1

I don't know about using awk instead of grep, but this works for me:

tail -f file.log | grep -Ev '(ELB|Pingdom|Health)'

EDIT: As dmourati and Caleb pointed out, you could also use egrep instead of grep -E for convenience. On some systems this this will be an link to the same binary, in others a copy of it supplied by the grep package. Either way it lives as an alternative to the -E switch. However, according to the GNU grep man page:

[…]two variant programs egrep and fgrep are available. egrep is the same as grep -E. fgrep is the same as grep -F. Direct invocation as either egrep or fgrep is deprecated, but is provided to allow historical applications that rely on them to run unmodified.

Since they are synonymous commands, it comes down to preference unless you don't have egrep at all. However for forward compatibility it is recommended to use the grep -E syntax since the other method is officially deprecated.

Solution 2

Try piping it to egrep with a pipe separated lists of words you want to filter out:

tail -f log_file | egrep -v 'ELB|Pingdom|Health'

Note that using parenthesis around the list of matches is optional. Since the | is treated as a logical OR operator by grep whether it occurs as part of a sub-group or not. '(ELB|Pingdom|Health)' would function exactly the same. For some, the syntax may be more obvious; I find it easier to type without since I can switch from a single match to a list of possible matches without going back to add the parenthesis.

For extra credit, it's worth mentioning that multitail does ninja foo when it comes to filtering output. For example you could filter for your words like this:

multitail -e ELB -e Pingdom -e Health -f log_file

You could also use it to color or otherwise highlight the output instead of just filtering it.

EDit: See DTests answer and the comments for the full explanation of how egrep is just a deprecated alternate way to fire off grep -E.

Solution 3

tail -f /path/to/log | egrep -v 'ELB|Pingdom|Health'

Solution 4

Why do you want to log this information?

  • Is it strictly for archival?
  • Do you want to conditionally execute different scripts depending on different keywords or patterns in the log files?

If you want to have scripted behavior depending on the content of the log files, you may wish to do your filtering using Expect. ( http://en.wikipedia.org/wiki/Expect ) Expect is a Tcl extension but There is also a Python version of Expect.

Expect gives you this powerful flexible switch like statement that lets you specify different behaviors conditionally depending on the states, or patterns present in your input stream. For example:

expect {  
    "password:" {  
        send "password\r"  
    } 
    "yes/no)?" {  
        send "yes\r"  
        set timeout -1  
    }  
    timeout {  
        exit  
    }   
    -re . {  
        exp_continue  
    }  
    eof {  
        exit  
    }  
}

So you specify patterns in the expect statement, and you specify different behaviors, and you can wrap the whole thing in a loop, and you can easily write very powerful filters that also write portions of your input to different files, or drop it altogether, or take actions and run other scripts depending on what is in your input.

So, it comes down to why are you trying to filter your log files, to take action on log input, or just for archival reasons?

Share:
35,598

Related videos on Youtube

Alex
Author by

Alex

Updated on September 18, 2022

Comments

  • Alex
    Alex over 1 year

    I want to tail -f my logs. However, I want to filter out everything that has the words:

    "ELB", "Pingdom", "Health"

  • bbaja42
    bbaja42 almost 13 years
    shouldn't expression end with ('), not with (")?
  • Caleb
    Caleb almost 13 years
    Yes thanks that was a typo. For future referance since stack exchange sites function like wikis, that is the kind of thing you can just fix.
  • Sirex
    Sirex almost 13 years
    I thought the edit had to be more than 6 chars ?
  • MPi
    MPi almost 13 years
    Plus one for the reference to Expect, which I used quite a long time ago and had completely forgotten about.
  • Caleb
    Caleb almost 13 years
    If you don't have high rep, yes there is a 6 character minimum, but in this case the 1 character is super important. You can force the change through by adding an HTML comment to the body. The characters will count towards the limit and you can note why you are making the change.
  • Caleb
    Caleb almost 13 years
    Does using grep -E instead of egrep warrant a duplicate answer?
  • Derek Downey
    Derek Downey almost 13 years
    @Caleb I don't see why not. There's more than one way to skin a cat, and this site allows for the ability to list all
  • Marcin
    Marcin almost 13 years
    Yes, but they're merely symlinks to each other, not two different programs with some functional overlap. So wouldnt that be more of an 'addendum' (read: comment), not a full fledged answer? I've had downvotes for lesser offences...
  • Caleb
    Caleb almost 13 years
    @DTest: By rights dmourati actually beat me to the punch by a couple of seconds and although he didn't explain the reasoning he deserves some credit here. You were distinctly late to the party since we both had at least two upvotes before you came in. Changing the syntax from a symlinked binary to an argument is usually something you would use a comment for, not a separate answer. If you want to skin the cat differently use sed, awk, perl, multitail or ninja_foo.
  • user9517
    user9517 almost 13 years
    egrep isn't a symlink on the Ubuntu 9 & 10, Solaris 10 and OpenSolaris systemsI have to hand although it is on a Centos 5.
  • Caleb
    Caleb almost 13 years
    I saw your meta post. I'd already read the couple dozen threads on SO-meta before commenting. It's not a big deal. I almost deleted my answer when I saw it came out the same as dmourati's so that he could get the points and encourage him to stick around, and only opted to leave it because I explained what I was doing to a question that obviously is a newbie unix user. I'm over counting my rep, but what I am is an edit natzi. (Note how many times I've edited posts to fix people's typos). I think you could add value by explaining why someone would use egrep -E over egrep.
  • Caleb
    Caleb almost 13 years
    Also, I'm sorry if my tone was harsh in my first comment. I could have been more constructive and explained the difference myself. It didn't occur to me that you didn't know, but I should have know the original asker wouldn't know. Forgive me.
  • Derek Downey
    Derek Downey almost 13 years
    No worries. I learned about egrep out of this discussion. Any time I learn something, I call it a plus.
  • dmourati
    dmourati almost 13 years
    Not a big deal. For what it's worth I learned the grep -E is the preferred invocation at that my suggestion of egrep is actually deprecated.
  • sidewinderguy
    sidewinderguy about 7 years
    @Caleb thanks so much for the multitail suggestion it is awesome!! Can't believe I went for so long in my life without it.