Zookeeper refuses Kafka connection from an old client

Question

I have a cluster configuration using Kubernetes on GCE, I have a pod for zookeeper and other for Kafka; it was working normally until Zookeeper get crashed and restarted, and it start refusing connections from the kafka pod:

Refusing session request for client /10.4.4.58:52260 as it has seen zxid 0x1962630

The complete refusal log is here:

2017-08-21 20:05:32,013 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /10.4.4.58:52260
2017-08-21 20:05:32,013 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@882] - Connection request from old client /10.4.4.58:52260; will be dropped if server is in r-o mode
2017-08-21 20:05:32,013 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@901] - Refusing session request for client /10.4.4.58:52260 as it has seen zxid 0x1962630 our last zxid is 0xab client must try another server
2017-08-21 20:05:32,013 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for client /10.4.4.58:52260 (no session established for client)

Solution 1

Because the kafka maintain a zookeeper session which remember the last zxid it has seen. So when the zookeeper sevice go down and come again, the zk's zxid begin from a smaller value. and ZKserver think the kafka has seen a bigger zxid, so it refuse it.

Have a try to restart the kafka.

Solution 2

For the record, I had this problem and all my kafka were off.
But, my kafka-manager was still up and listening on zookeepers. Turning it off resolved the issue.

Solution 3

Related to the answer from @GuangshengZuo.... Steps

1. Stop any residual zookeeper instances - zookeeper-server-stop.bat
2. Start a fresh zookeeper- zookeeper-server-start.bat .\config\zookeeper.properties This will do

Comments

[sameera sy]

How did you turn off the manager? turning off the kafka server doesn't seem to working