Elasticsearch 503 error when checking server status

Question

I've been using elasticsearch as a search engine for my Rails application, however it stopped working properly due to a reason i can't understand. When making a curl request to elasticsearch server i get a 503 error.

curl -XGET http://localhost:9200
{
  "ok" : true,
  "status" : 503,
  "name" : "Killpower",
  "version" : {
    "number" : "0.90.3",
    "build_hash" : "5c38d6076448b899d758f29443329571e2522410",
    "build_timestamp" : "2013-08-06T13:18:31Z",
    "build_snapshot" : false,
    "lucene_version" : "4.4"
  },
  "tagline" : "You Know, for Search"
}

I've tried to solve the problem with reloading elasticsearch service and installing a new version without any success.

Here is what I found in logs

[2013-09-03 12:31:45,320][INFO ][node                     ] [Killpower] version[0.90.3], 
pid[4222], build[5c38d60/2013-08-06T13:18:31Z]
[2013-09-03 12:31:45,321][INFO ][node                     ] [Killpower] initializing ...
[2013-09-03 12:31:45,327][INFO ][plugins                  ] [Killpower] loaded [], sites []
[2013-09-03 12:31:47,248][INFO ][node                     ] [Killpower] initialized
[2013-09-03 12:31:47,248][INFO ][node                     ] [Killpower] starting ...
[2013-09-03 12:31:47,313][INFO ][transport                ] [Killpower] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/172.20.46.32:9300]}
[2013-09-03 12:31:51,443][INFO ][discovery.zen            ] [Killpower] master_left [[Amphibion][IMB4uACSTyOx3MO2u-FsWg][inet[/fe80:0:0:0:52e5:49ff:fec2:9718%2:9300]]], reason [do not exists on master, act as master failure]
[2013-09-03 12:31:51,446][INFO ][discovery                ] [Killpower] elasticsearch/iavpGrMtRHmWLHMTNpscGQ
[2013-09-03 12:31:51,483][INFO ][http                     ] [Killpower] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/172.20.46.32:9200]}
[2013-09-03 12:31:51,484][INFO ][node                     ] [Killpower] started
[2013-09-03 12:31:54,712][WARN ][transport.netty          ] [Killpower] exception caught on transport layer [[id: 0xa929f24d, /127.0.0.1:50456 => /127.0.0.1:9300]], closing connection

After that also follows a long java stacktrace. How can i fix the problem?

Solution 1

There could be another instance in your local network with elasticsearch server up and running. Since shards in elasticsearch are working from the box and enabled by default there could be a conflict of master node. Please review your elasticsearch.log file. If there is something like

{"error":"MasterNotDiscoveredException[waited for [30s]]","status":503} 

You should go to your config file /etc/elasticsearch/elasticsearch.yml and add this line:

discovery.zen.ping.multicast.enabled: false

Solution 2

in addition check value of

discovery.zen.ping.unicast.hosts

because it is necessary to discover elastic nodes