Load Balancer with Zookeeper

Question

I'm trying to create a Load Balancer to be in front of a Zookeeper 3.4.6 cluster. When I do that the cluster works well but an exception is thrown:

WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)

It means that Zookeeper is understanding Load Balancer as a client and it's tryong to stablish a connection with it. But the Load Balancer just pings TCP 2181 and comes out.

Solution

You are trying to use a load balancer between your ZooKeeper cluster and clients?

When you give your clients a ZooKeeper connection string in the form of multiple endpoints like this: "server1,server2,server3...", the clients will pick one of the servers and switch over in case of failure. This way, if all your clients have the same ZooKeeper endpoints string, you will end up with a balanced pool.

If you put a standard load balancer between the clients and the server, it can cause failures like this. A load balancer doesn't play well with the way ZooKeeper expects its clients to behave. A client needs to maintain an open TCP connection to a specific server it has a session on, sending periodic heartbeats.

There are certain limitations to the way ZooKeeper clients load balance themselves (e.g. connections won't rebalance in case of server restarts), but fixing these limitations would require a ZooKeeper protocol aware load balancing logic, probably as part of the client implementation.