Foward slash in kibana 3 query

Question

I'm trying to add a query that will match a request that ends with a slash, like this one:

n.n.n.n - - [16/Oct/2013:16:40:41 +0100] "GET / HTTP/1.1" 200 25058 "-" "Mozilla/5.0 (iPad; CPU OS 7_0_2 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A501 Safari/9537.53"

I'm using the Lucene query type.

If my query is set to *, I see the event.

If I set it to request:"css", I see CSS requests, as expected.

However, all of the following yield no results:

• request:"/"
• request:"\/"
• request:"\\/"

I tried a Lucene regular expression, with no luck:

• request:/\//

I note that someone else is getting what appears to be a similar issue, although that's on Kibana 2: https://github.com/rashidkpc/Kibana/issues/401

How can I query for requests that end with a / character?

Solution

I have managed to work around my problem by adding a field before records are output to elasticsearch.

In my indexer.conf file, I have added this code:

filter {
  if [request] =~ /\/$/ {
    mutate {
      add_field => {
        'file_type' => 'html'
      }
    }
  }
}

I can now pick out the records that I'm interested in with the query file_type:"html".

This may actually be a better way of doing it, since there is warning about using leading wildcards here:

Allowing a wildcard at the beginning of a word (eg "*ing") is particularly heavy, because all terms in the index need to be examined, just in case they match.

source:http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#_wildcards

So, I'm probably going to add tests for images, JavaScript, CSS, etc..

Comments

[Stefan Förster]

If you are using logstash 1.3.x, every field will come with a virtual '.raw' field, in your case probably 'request.raw'. This field is not analyzed and you should be able to search for request.raw:"/". See also: elasticsearch.org/blog/logstash-1-3-1-released

[Stefan Förster]

request.raw:/.*\// perhaps?

[G Mawr]

@StefanFörster No, that doesn't do it. I'm having difficulty making regular expression queries work at all, even those not restricted to particular fields. When using regular expression, these all return results: .*, ., .., \s{2}. These do not return results: \/, \s{3}, GET. I think the _all field is not being populated, but am unsure how to check or fix that.

[G Mawr]

I haven't defined any mapping. I just followed the instructions at logstash.net/docs/1.2.1/tutorials/getting-started-centralize‌​d. I have found the elasticsearch.yml file, but haven't made any changes to it. I did add a term panel, and it does indeed appear that the request values are being split into multiple terms, dropping the /.