How to remove an event from logstash?

Question

I have a line in my log files that literally just have a semi colon in them. I am assuming it is attached to the previous line. Logstash is constantly printing them, and I want to drop these when ever there is a line that begins with a ;.

This is what logstash prints:

"message" => ";/r"
"@version" => "1"
"@timestamp" => 2014-06-24T15:39:00.655Z,"
"type" => "BCM_Core",
"host => XXXXXXXXXXX",
"Path => XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"tags" => [
[0] "_grokparsefailureZ"
],
"BCM_UTC_TIME" =>"2014-06-24%{time}Z"

I've attempted to use multiline to append to previous line so logstash would stop printing:

   multiline{
    type => "BCM_Core"
    pattern => "\;"
    negate => true
    what => "previous"
}

but logstash is still printing them out. How can I make logstash drop it?

Solution

Just use a drop filter to drop any line that starts with ;:

filter {
   if ([message] =~ "^;") {
      drop {}
  }
}

Although based on your output, it really ;/r not ;\r, so you might need to adjust if your output is not just an example.

You can also just drop anything that fails to grok:

if "_grokparsefailure" in [tags] { drop {} }

Comments

[juicymango]

why is it /^;$/? Shouldn't it just be /;? And yes my output was ";\r", so what should I should the filter to?

[Alcanzar]

The $ matches end of line (so basically saying only ignore lines with just a ; to avoid the possibility of dropping too much). You could probably just do /^;/ -- so anything starting with ;... and oops my syntax is wrong for logstash.. fixed

[juicymango]

Just one other question. Am I allow to use grok more than once in the filter? Will that cause grokparsefailure since the lines can match all the grok filters?

[Alcanzar]

you can use multiple grok's, but if any of them fail you'll get the _grokparsefailure, so you can put if [message] =~ "some pattern" { grok {} } around them to prevent that sort of thing.