Drop log messages containing a specific string
Solution 1
To drop the message that does not contain the string xyz
:
if ([message] !~ "xyz") {
drop { }
}
Your grok pattern is not grabbing the date part of your logs.
Once you have a field from your grok pattern containing the date, you can invoque the date filter on this field.
So your grok filter should look like this:
grok {
match => {
"message" => '%{SYSLOG5424SD:loglevel} <%{JAVACLASS:job}> %{TIMESTAMP_ISO8601:Date} %{GREEDYDATA:content}'
}
}
I added a part to grab the date, which will be in the field Date
. Then you can use the date filter:
date {
match => [ "Date", "YYYY-mm-dd HH:mm:ss,SSS" ]
locale => en
}
I added the ,SSS
so that the format match the one from the Date
field.
The parsed date will be stored in the @timestamp
field, unless specified differently with the target
parameter.
Solution 2
to check if your message contains a substring, you can do:
if [message] =~ "a" {
mutate {
add_field => { "hello" => "world" }
}
}
So in your case you can use the if to invoke the drop{} filter, or you can wrap your output plugin in it.
To parse a date and write it back to your timestamp field, you can use something like this:
date {
locale => "en"
match => ["timestamp", "ISO8601"]
timezone => "UTC"
target => "@timestamp"
add_field => { "debug" => "timestampMatched"}
}
This matches my timestamp in:
- Source field: "timestamp" (see match)
- Format is "ISO...", you can use a custom format that matches your timestamp
- timezone - self explanatory
- target - write it back into the event's "@timestamp" field
- Add a debug field to check that it has been matched correctly
Hope that helps,
Artur
Karup
Talk is cheap. Show me the code. An imaginary dunker and a procrastinating perfectionist. #SOreadytohelp
Updated on July 13, 2022Comments
-
Karup almost 2 years
So I have log messages of the format :
[INFO] <blah.blah> 2016-06-27 21:41:38,263 some text [INFO] <blah.blah> 2016-06-28 18:41:38,262 some other text
Now I want to drop all logs that does not contain a specific string "xyz" and keep all the rest. I also want to index timestamp.
grokdebug is not helping much.
This is my attempt :
input { file { path => "/Users/username/Desktop/validateLogconf/logs/*" start_position => "beginning" } } filter { grok { match => { "message" => '%{SYSLOG5424SD:loglevel} <%{JAVACLASS:job}> %{GREEDYDATA:content}' } } date { match => [ "Date", "YYYY-mm-dd HH:mm:ss" ] locale => en } } output { stdout { codec => plain { charset => "ISO-8859-1" } } elasticsearch { hosts => "http://localhost:9201" index => "hello" } }
I am new to grok so patterns above might not be making sense. Please help.