elasticsearch - how to copy data to another cluster

Question

How can I get an elasticsearch index to a file and then insert that data to another cluster? I want to move data from one cluster to another but I can't connect them directly.

Solution 1

If you no need to keep _id the same and only important bit is _source you may use logstash with config:

input { //from one cluster } output { //to another cluster }

here is more info: http://www.logstash.net/docs/1.4.2/

Yes it's method is weird, but I tried it for instant data transfer between clusters index by index and it is working as a charm (of course if you no need to keep _id generated by elasticsearch)

Solution 2

There is script which will help you to backup and restore indices from one cluster to another. i didn't tested this but may be it will fix your needs. check this Backup and restore an Elastic search index

And you can also use perl script to copy index from one cluster to another (or the same cluster).

check this link clintongormley/ElasticSearch.pm

Comments

[colidyre]

links are dead. Please update links or delete answer.

[kiruba]

@colidyre check this link but its old and i think they are strongly recommand to use official python api. github.com/eriky/ESClient. check this also my forked version has copy script github.com/kirubar/ESClient