hbase connection refused

Question

I am new to HBase and Hadoop. I have completely setup HBase and started perfectly. Now when I tried to connect from p1 to HBase (where HBase setup is on p2) using a Java client, then it is throwing a strange exception.

12/04/17 14:36:37 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=192.168.15.20:2181 sessionTimeout=180000 watcher=hconnection
12/04/17 14:36:38 INFO zookeeper.ClientCnxn: Opening socket connection to server /192.168.15.20:2181
12/04/17 14:36:38 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
12/04/17 14:36:39 INFO zookeeper.ClientCnxn: Opening socket connection to server hbase.local/192.168.15.20:2181
12/04/17 14:36:39 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
12/04/17 14:36:41 INFO zookeeper.ClientCnxn: Opening socket connection to server hbase.local/192.168.15.20:2181
12/04/17 14:36:41 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
12/04/17 14:36:43 INFO zookeeper.ClientCnxn: Opening socket connection to server hbase.local/192.168.15.20:2181
12/04/17 14:36:43 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
12/04/17 14:36:44 INFO zookeeper.ClientCnxn: Opening socket connection to server hbase.local/192.168.15.20:2181
12/04/17 14:36:44 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
12/04/17 14:36:45 INFO zookeeper.ClientCnxn: Opening socket connection to server hbase.local/192.168.15.20:2181
12/04/17 14:36:45 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
12/04/17 14:36:47 INFO zookeeper.ClientCnxn: Opening socket connection to server hbase.local/192.168.15.20:2181
12/04/17 14:36:47 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
12/04/17 14:36:48 INFO zookeeper.ClientCnxn: Opening socket connection to server hbase.local/192.168.15.20:2181
12/04/17 14:36:48 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
12/04/17 14:36:49 INFO zookeeper.ClientCnxn: Opening socket connection to server hbase.local/192.168.15.20:2181
12/04/17 14:36:49 INFO zookeeper.ZooKeeper: Session: 0x0 closed
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:155)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1002)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:304)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:295)
    at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:157)
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:90)
    at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:1258)
    at HBaseDemo.main(HBaseDemo.java:55)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:903)
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
    ... 7 more
12/04/17 14:36:49 INFO zookeeper.ClientCnxn: EventThread shut down
Execution script exiting 

---

/04/17 17:41:35 INFO zookeeper.ZooKeeper: Client environment:user.name=xyz
12/04/17 17:41:35 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/xyz
12/04/17 17:41:35 INFO zookeeper.ZooKeeper: Client environment:user.dir=/opt/xyz/hbase-sample
12/04/17 17:41:35 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=192.168.15.169:2181 sessionTimeout=180000 watcher=hconnection
12/04/17 17:41:35 INFO zookeeper.ClientCnxn: Opening socket connection to server /192.168.15.169:2181
12/04/17 17:41:35 INFO zookeeper.ClientCnxn: Socket connection established to hbase.xyz.com/192.168.15.169:2181, initiating session
12/04/17 17:41:35 INFO zookeeper.ClientCnxn: Session establishment complete on server hbase.xyz.com/192.168.15.169:2181, sessionid = 0x136c00d59ac0009, negotiated timeout = 180000
12/04/17 17:41:36 INFO client.HConnectionManager$HConnectionImplementation: getMaster attempt 0 of 1 failed; no more retrying.
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:604)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:360)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:94)
at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:1258)
at HBaseDemo.main(HBaseDemo.java:55)
HBase is not running!
Execution script exiting 

Solution 1

I have found the solution.

By just removing localhost entry from my /etc/hosts. Now my localhost entry is like 192.169.19.50 [which is my hbase IP] = localhost instead of 127.0.0.1 localhost.

Solution 2

The accepted answer here might work, but only as a side effect.

The problem is that HBase is opening a port listening on localhost (i.e., on loopback) rather than on the external facing interface. Removing the localhost entry in /etc/hosts forces HBase to bind to the external interface -- but only because it can't find localhost. This is likely to become a problem when some daemon wants to legitimately listen on or connect to localhost (via lo) alone.

You can test that you have this problem by using telnet to connect to port 60000 via localhost or the IP assigned to the machine. It's likely you'll be able to connect via localhost (because that's where HBase is bound) and not to the IP.

Better to use your machine name in your HBase conf.

Solution 3

Assuming that you have instantiated Configuration for your HBase, set hbase master, the zookeeper quorum and the znode parent. Kinda like this-

import org.apache.hadoop.hbase.HBaseConfiguration;

public class CLASS_NAME{
    Configuration configuration = new HBaseConfiguration.Create();
    configuration.set(hbase master, "");
    configuration.set(zookeeper quorum, "");
    configuration.set(znode parent, "");  
}

Comments

[sunskin]

khan - i have this error when i am running hbase, hadoop, nutch on the same server in pseudo mode. Do I need to edit my /etc/hosts file?

[user1455836]

A bit doubtful solution. You should possibly better modify hbase.zookeeper.quorum property either on client or server side.

[Eric Walker]

@user1455836's suggestion, and flushing the DNS cache and restarting (pure superstition?), worked for me. For my Mac, I added an entry for hbase.zookeeper.quorum to /usr/local/Cellar/hbase/1.0.1/libexec/conf/hbase-site.xml with a value of <hostname>.local. Zookeeper is now alive again with all of its blessings, glory and user friendliness.

[joaolsouzajr]

When to configure a cluster to set property "hbase.master" in hbase-site.xml with your hostname or IP, example: code <property> <name>hbase.master</name> <value>192.168.1.111:16000</value> </property>