Query solr cluster for state of nodes

Question

I'm trying to tweak our system status check to see the state of the Solr nodes in our SolrCloud. I'm facing the following problems:

We send a query to each of the Solr nodes separately. If we get a response and the status of the response is 0, we assume the node is running. Unfortunately, we've seen cases in which the node is recovering or even down and select queries are still handled.

In hope to prevent this, we've added a check which sends a ping request to solr. If the status returned by this is request reads 'OK' we assume the node is up. Unfortunately even with this request, if the node is recovering or down, this check won't fail.

My question is: What is the correct way to check the status of a node in SolrCloud?

Solution 1

If you are using a SolrCloud, it's recommended to maintain an explicit zookeeper ensemble as well. Because zookeeper ensemble maintains the SolrCloud's current status of each node and each shard wise. This status is actually get reflected from the SolrCloud admin window.

1. Go to the Admin window. Click on "Cloud".
2. Then click on "Tree" to get a tree view of your SolrCloud architecture.
3. Click /clusterstate.json to view the SolrCloud status.

This (clusterstate.json) json file holds the SolrCloud status information. Now if you are running an explicit zookeeper ensemble, following are the steps to get SolrCloud status.

1. Go to the path "zookeeper/installation/directory/bin"
2. Execute ./zkCli.sh -server ZK_IP:ZK_PORT (E.g ./zkCli.sh -server localhost:2181)
3. Execute get /clusterstate.json

You'll find the SolrCloud status.

Note : ZK_IP - The HOST IP where zoopeeper is running. ZK_PORT - Zookeeper's client port.

Solution 2

You actually don't want /clusterstate.json - as this only covers the case where collections are already present. From ZooKeeper you need /live_nodes

Because Zookeeper is the authority for what Solr Nodes are members of the Solr cloud cluster, it follows that you should go to it first, to discover what members are accessible. This is how all Solr cloud clients work, and probably is the best way to approach the problem.

/live_nodes contains a file for each live Solr node, regardless of what collections exist or where the replicas are located.

Once you have resolved /live_nodes... you can call clusterstatus on any Solr instance with the address and port from one of the live-nodes.

http://localhost:8983/solr/admin/collections?action=clusterstatus&wt=json

clusterstatus provides a detailed overview of Solr nodes, collections, replicas, etc. Everything you would want to know.

As a final note, it's very wise to set SOLR_HOST inside of solr.in.sh configuration (/etc/default/solr.in.sh) - by default 'localhost' is used to reference the solr node. Setting this value to the public address you want the Solr node identified by will prevent ZooKeeper from returning the address "localhost" to clients when attempting to reach a Solr Node.

Comments

[nir]

What to check in clusterstate.json that says all nodes(solr servers) are running and healthy?

[SimplyInk]

I second checking /live_nodes before /clusterstate.json.

[SimplyInk]

I had stale /clusterstate.json data where a Solr node was "state":"active" but was actually down. /live_nodes is more sensitive as it is an Ephemeral Znode created by the Solr node's ZooKeeper connection. ZooKeeper (ensemble) will automatically delete this Znode when the Solr node's ZooKeeper connection timeout.