HBase standalone failed to connect (fail to create table)

Question

I am trying to deploy Hbase in standalone mode following this article: http://hbase.apache.org/book.html#quickstart. The version is 0.92.1-cdh4.1.2

But I am getting these errors when try to create a table:

Error message:

    13/04/01 14:07:10 ERROR zookeeper.RecoverableZooKeeper: ZooKeeper exists failed after 3 retries
13/04/01 14:07:10 WARN zookeeper.ZKUtil: hconnection Unable to set watcher on znode /hbase/master
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:226)
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:580)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:569)

Output log:

13/04/01 14:06:39 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=180000 watcher=hconnection
13/04/01 14:06:39 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 10231@localhost
13/04/01 14:06:39 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration)
13/04/01 14:06:39 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
13/04/01 14:06:39 WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid

My configurations:

1. Added JAVA_HOME in hbase-env.sh

2. hbase_site.xml

   <name>hbase.rootdir</name>
   
   <value>file:///home/hadoop/data</value>
   

   <name>hbase.zookeeper.property.dataDir</name>
   
   <value>file:///home/hadoop/zookeeper</value>
   

I have tried to modify /etc/hosts, which looks likes this (oracle is the hostname):

127.0.0.1   localhost oracle

or

[server ip]   localhost oracle

But these do not work for me.

My exact codes are:

[usr@oracle bin]$ ./start-hbase.sh 
starting master, logging to /home/hadoop/hbase-0.94.6/bin/../logs/hbase-usr-master-oracle.out
[usr@oracle bin]$ hbase shell
13/04/01 14:57:55 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.92.1-cdh4.1.2, rUnknown, Thu Nov  1 18:02:07 PDT 2012

hbase(main):001:0> create 'test','cf'

Thanks in advance!

Solution 1

Looks like you are missing hbase.zookeeper.quorum in your configuration. Please check this HBase guide chapter, it may help: http://hbase.apache.org/book/zookeeper.html

Also please check zookeeper is started on right port and (what happens more often) correct IP interfaces.

Solution 2

I'm not sure if you're on Mac, but I had this problem as well. It turns out that it was because a .DS_STORE file was in a folder of -ROOT- region server and causing a connect exception. Very strange!

Solution 3

In HBase 1.2.3 I've got almost the same error "ZooKeeper exists failed after 4 retries" in the standalone mode. It was caused by running ./start-hbase.sh without having permissions to connect to the port 2181. The solution turned out to be really simple:

sudo ./start-hbase.sh

Just in case, the configuration of hbase-site.xml is:

<configuration>
   <property>
      <name>hbase.rootdir</name>
      <value>file:///home/hadoop/HBase/HFiles</value>
   </property>
   <property>
      <name>hbase.zookeeper.property.dataDir</name>
      <value>/home/hadoop/zookeeper</value>
   </property>
</configuration>

Solution 4

Something might be related to your configuration. At first, I download HBase tar(http://hbase.apache.org/book/quickstart.html) to run HBase. I didn't follow the link to do any configuration, just download, start HBase, run "HBase shell". But when creating a table, it gave me error like you mentioned. And then, I search HBase installation on Mac, since I use Mac. I adopt to use "brew install hbase" to install HBase(http://chase-seibert.github.io/blog/2013/02/01/getting-starting-with-hbase-and-pig.html). After finished installation, I re-create the table. Success.

So re-consider your error, it must be linked with your JAVA_HOME or other configurations on HBase. Finding the answer from this way might help you.

Solution 5

The error "ERROR zookeeper.RecoverableZooKeeper: ZooKeeper exists failed after 3 retries" most probably indicates you don't have zookeeper running - Before staring the HBase shell you can verify if the Zookeeper quorum is up, using:

$ jps

the command will list all the java processes on the machine i.e. the probable output has to be - for the standalone HBase setup you're trying to setup (ignore the numbers in the left column i.e. pid):

62019 Jps
61098 HMaster        
61233 HRegionServer     
61003 HQuorumPeer

If your output doesn't have the HQuorumPeer - indicates that the zookeeper isn't running.

Zookeeper is required for HBase cluster - as it manages it.

---

Solution:

In your HBase directory, first stop the HBase:

$ ./bin/stop-hbase.sh

If you're trying to work out the 'standalone HBase' example - stick to the minimum conf provided in the example:

<configuration>
  <property>
    <name>hbase.rootdir</name>
    <value>file:///home/adio/workspace/hadoop/hbase/directories/hbase</value>
  </property>
  <property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/home/adio/workspace/hadoop/hbase/directories/zookeeper</value>
  </property>
</configuration>

i.e. your conf/hbase-site.xml should have the above content.

Once set, now start the HBase again:

$ ./bin/start-hbase.sh

P.S. anyone if going through these steps still have the issue not resolved - leave your issue in the comment section.

---

Few relevant answers: 1 2 3

View more solutions

Comments

[pferrel]

can you explain what "a folder of -ROOT- region server" means. The default standalone config uses a version of ZooKeeper launched by Hbase itself. I get this error on the mac with the standalone Hbase.

[pferrel]

Using the standalone hbase I get the same error so zookeeper is launched by hbase as needed. When the table is created hbase can't launch zookeeper because it says "ZooKeeper exists failed after 3 retries". Which seems to say that hbase thinks zookeeper exists but the connection fails. I look at jps -l and see no zookeeper so why does it think it exists?

[pferrel]

No luck with that myself. Also took out all ipv6 references to localhost in /etc/hosts

[pferrel]

The first time launch of a standalone hbase will create the folder according to the hbase quickstart. I've tried with a clean dataDir and with a non-existent one, same error.

[Eric Walker]

When I do this, it fixes the ZK error, but I also lose my local HBase tables.

[zuazo]

Please, be more careful when formatting your answer.

[joh-mue]

this actually solved it for me somehow. It was driving me nuts, very strange that this is not in the instructions as it does seem necessary in macOS Big Sur.