How to grep for lines which contain particular words in a log file?

Question

I have a big log file which I am trying to scan it for a particular words. In general, I will have few words which I need to grep on my big log file and print out the line which contains those words.

I know how to do simple grep on a file. Suppose if my file name is abc.log and I need to find a line which contains word "hello" then I always do it like this and it prints out the line for me.

grep -i "hello" abc.log

But I don't know how to do the grep for combination of words. Meaning I would have list of words and I will scan my abc.log file for all those words and I will print out the lines which contains those words individually.

#!/bin/bash

data="hello,world,tester"

# find all the lines which contains word hello or world or tester

So in my above shell script I will split my data variable and look for hello word in abc.log so any line which contains hello word, I will print it out and similarly with world and tester as well.

I am trying to make this pretty generic so that I just need to add my list of words in the data variable without touching the actual logic of grepping the logs.

Solution 1

If you store your patterns in a file, one per line, you can use grep -f file-with-patterns file-to-search.log

From the man page:

   -f FILE, --file=FILE
          Obtain  patterns  from  FILE,  one  per  line.   The  empty file
          contains zero patterns, and therefore matches nothing.   (-f  is
          specified by POSIX.)

---

Edit 2018:

Since I wrote this, I have become aware of the following interesting edge cases:

• You can read the list of patterns from pipe using -f - (if you don't need stdin, i.e. you specified files on grep's command line) or -f <() (in any case)
• grep's performance starts to fail badly if hundreds of patterns are passed. If your use case is that insane, consider generating and immediately executing a sed (or some other language) script, although this could potentially have problems with overlapping patterns.

Solution 2

I would use a regular expression, like this:

grep -E 'hello|world|tester' abc.log

Solution 3

Apart from bruchowski's answer, you can also use:

grep -i -e "hello" -e "world" -e "tester" abc.log

OR

grep 'hello\|world\|tester' abc.log

OR

egrep 'hello|world|tester' abc.log

Comments

[john]

thanks o11c. In my case patterns might grow a lot so file approach is good but I wanted to add one more thing in this - don't print out the line if it contains a particular word. How will I do this with your current approach?

[o11c]

@user2809564 pipe the first grep through grep -v wordtoexclude (or -e or -f, etc)

[john]

Thanks a lot bruchowski. It works fine. In my case I can have more than three patterns so I was thinking to either store it in a variable just like I have shown in my question and then split that variable and start grepping it or either store it in a file as well. Do you think any better way of making this generic in a shell script?

[o11c]

@user2809564 since there is a limit to the length of command-line arguments, this approach will eventually fail where the file one won't. I'm not sure whether your data set is that big though.

[bruchowski]

@user2809564 either one is a good approach. If you don't plan on having too many strings to search then I might go with this inline approach and just join an array of terms on |, otherwise @o11c's answer would work well for you

[john]

Thanks. It might be possible that I can have multiple patterns as well to exclude so is there any way to make one file which has patterns to include and pattern to exclude and then use this file to do the grep?

[o11c]

@user2809564 you'll separate files for includes and excludes: grep -f include-file search-files... | grep -v exclude-file

[john]

got it. one last thing, suppose if I am looking for a line which has word hello in it but my word to exclude is test so the line which contains hello if it also contains test then that line will get excluded right?

[Damodar Bashyal]

grep -Rie "80" ./ helped me find recursive from the current folder.

[Damodar Bashyal]

grep -Rie "listen.*80" ./ worked as a regular expression search. Awesome!