Elasticsearch: No handler for type [keyword] declared on field [hostname]

Question

I get above Mapper Parsing Error on Elasticsearch when indexing log from filebeat.

I tried both Filebeat -> Elasticserach and Filebeat -> Logstash -> Elasticsearch approach.

I have followed their own documentations, I installed filebeat template as per instructed and verified from Loading the Index Template in Elasticsearch | Filebeat Reference

My elasticsearch is normally working fine with my other data indexing and I tested them on Kibana. Its an official docker Docker Hub | Elasticsearch installation.

Googled a lot without any luck so, any help is appreciated.

UPDATE 1:

ES version: 2.3.3 (I believe latest one)

Template file is the default shipped with filebeat.

{
  "mappings": {
    "_default_": {
      "_all": {
        "norms": false
      },
      "dynamic_templates": [
        {
          "fields": {
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "match_mapping_type": "string",
            "path_match": "fields.*"
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "beat": {
          "properties": {
            "hostname": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "name": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        "input_type": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "message": {
          "norms": false,
          "type": "text"
        },
        "offset": {
          "type": "long"
        },
        "source": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "type": {
          "ignore_above": 1024,
          "type": "keyword"
        }
      }
    }
  },
  "order": 0,
  "settings": {
    "index.refresh_interval": "5s"
  },
  "template": "filebeat-*"
}

UPDATE 2: You are right, see

#/usr/share/filebeat/bin/filebeat --version filebeat version 5.0.0-alpha2 (amd64), libbeat 5.0.0-alpha2

Though this is posting apache log to logstash. But I can't get this vhost_combined log in right format

sub1.example.com:443 1.9.202.41 - - [03/Jun/2016:06:58:17 +0000] "GET /notifications/pendingCount HTTP/1.1" 200 591 0 32165 "https://sub1.example.com/path/index?var=871190" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36"

"message" => "%{HOSTNAME:vhost}\:%{NUMBER:port} %{COMBINEDAPACHELOG}"

Solution

You cannot use "type": "keyword" with ES 2.3.3 since that's a new data type in ES 5 (currently in alpha3)

You need to replace all those occurrences with

"type": "string",
"index": "not_analyzed"

You need to use filebeat.template-es2x.json instead.

Comments

[rayhan]

I already figured this out from your last comment and things are coming nice on ES now. Filebeat official document should mention this very critical info as ES 2.3.3 is still official version in doc.

[Val]

Which version of filebeat do you have and how did you install it? I have the latest one, i.e. 1.2.3 and the filebeat-template.json file works perfectly well with ES 2.3.3, i.e. there's no "type": "keyword" in it.

[rayhan]

I installed filebeat from their official doc. They shipped the above mentioned es2 template along with filebeat.template.json in /etc/filebeat/ directory. I just deleted old template and put new template. This version doesn't have type keyword.

[Val]

Can you confirm you have installed Filebeat 1.2 and not Filebeat 5.0 ? They have different versions of their docs depending on which release you use. Right now the official packages are here and there's a single template file with no "keyword".

[Val]

Feel free to post another question with this new specific issue.

[rayhan]

Ok, any suggestion, do I need to downgrade filebeat? or this version is ok?

[Val]

The current official release is 1.2.3. The alpha2 version should not be used in production, so it depends on whether you want something stable or not :)

[rayhan]

The above answer solved my keyword data type issue by using filebeat.template-es2x.json template.