Elasticsearch: Auto Indices Deletion/Expiry

Question

I want to configure my elasticsearch 0.19.11 to delete indexes every 60s. My elasticsearch config has these 3 lines:

node.name: "Saurajeet"
index.ttl.disable_purge: false
index.ttl.interval: 60s
indices.ttl.interval: 60s

And its not working I have 2 default docs indexed. And would be expecting it to go after 60s

$ curl -XGET http://localhost:9200/twitter/_settings?pretty=true
{
  "twitter" : {
    "settings" : {
      "index.version.created" : "191199",
      "index.number_of_replicas" : "1",
      "index.number_of_shards" : "5"
    }
}

Also if i m trying to do the following it doesnot have any effect

$ curl -XPUT http://localhost:9200/twitter/_settings -d '
> { "twitter": {
>     "settings" : {
>       "index.ttl.interval": "60s"
>    }
>  }
> }
> '
{"ok":true}~/bin/elasticsearc
$ curl -XGET http://localhost:9200/twitter/_settings?pretty=true
{
  "twitter" : {
    "settings" : {
      "index.version.created" : "191199",
      "index.number_of_replicas" : "1",
      "index.number_of_shards" : "5"
    }
  }
}

I have indexes 2 documents and its still showing up after 1hr

$ curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '
{ 
    "user": "kimchy", 
    "postDate": "2009-11-15T13:12:00", 
    "message": "Trying out Elastic Search, so far so good?" 
}'
$ curl -XPUT 'http://localhost:9200/twitter/tweet/2' -d '
{ 
    "user": "kimchy", 
    "postDate": "2009-11-15T13:12:00", 
    "message": "Trying out Elastic Search, so far so good?" 
}'

WHAT DID I DO WRONG

P.S. I want to deploy this config with logstash. So any other alternative can be suggested. for scaling reasons i dont want this autopurge to be a script.

Solution 1

Finally Figured out myself. Upgraded elasticsearch version to 1.2.0. You can put in TTLs from the Mapping API. -> Put Mapping -> TTL.

Enabling TTL on type level on an index

$ curl -XPOST http://localhost:9200/abc/a/_mapping -d '
{
    "a": {
      "_ttl": {
        "enabled": true,
        "default": "10000ms"
      }
    }
}'

$ curl -XPOST http://localhost:9200/abc/a/a1 -d '{"test": "true"}'
$ $ curl -XGET http://localhost:9200/abc/a/a1?pretty
{
  "_index" : "abc",
  "_type" : "a",
  "_id" : "a1",
  "_version" : 1,
  "found" : true,
  "_source":{"test": "true"}
}
$ # After 10s
$ curl -XGET http://localhost:9200/abc/a/a1?pretty
{
  "_index" : "abc",
  "_type" : "a",
  "_id" : "a1",
  "found" : false
}

Note:

• Mapping applies to docs created after creation of mapping.
• Also mapping was created for type a. So if you post to type b and expect it expired on TTL, thats not gonna happen.

If you need to expire index, you can also create index level mappings during the create index to precreate indexes from your application logic.

Solution 2

I believe the indices.ttl.interval setting is only to tweak the cleanup process timing.

You would need to set the _ttl field for the index/type in order to expire it. It looks like this:

{
    "tweet" : {
        "_ttl" : { "enabled" : true, "default" : "60s" }
    }
}

http://www.elasticsearch.org/guide/reference/mapping/ttl-field/