Unable to start kafka with zookeeper (kafka.common.InconsistentClusterIdException)

Question

Below the steps I did to get this issue :

1. Launch ZooKeeper
2. Launch Kafka : .\bin\windows\kafka-server-start.bat .\config\server.properties

Also note that my OS is Windows

And at the second step the error happens :

ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) kafka.common.InconsistentClusterIdException: The Cluster ID Reu8ClK3TTywPiNLIQIm1w doesn't match stored clusterId Some(BaPSk1bCSsKFxQQ4717R6Q) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong. at kafka.server.KafkaServer.startup(KafkaServer.scala:220) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44) at kafka.Kafka$.main(Kafka.scala:84) at kafka.Kafka.main(Kafka.scala)

When I trigger .\bin\windows\kafka-server-start.bat .\config\server.properties zookeeper console returns :

INFO [SyncThread:0:FileTxnLog@216] - Creating new log file: log.1

How to fix this issue to get kafka running ?

EDIT See this answer following those steps solved my problem

Solution 1

I solved this issue on my machine (Windows 10) by deleting the meta.properties file found in the log directory.

The issue itself occured after I had stopped both the zookeeper and server and then restarted them.

Solution 2

Usually, this problem arises when kafka logs are stored in the persistent folder and zookeeper data in the temporary, or vice-versa. Then, after system restart, files that are stored in the temporary directory get cleaned and regenerated leading to the configuration mismatch.

To prevent this situation to happen again revise log.dirs kafka parameter in the server.properties and dataDir zookeeper parameter in zookeeper.properties and ensure that both point to the same type of directory (e.g. temporary or persistent). Only after that reset inconsistent data (if any) using for example method from Jacob's answer, i.e. remove meta.properties file, and restart zookeeper and kafka.

Solution 3

I encountered the same issue while running Kafka server on my Windows Machine. You can try the following to resolve this issue:

1. Open server server.properties file which is located in your kafka folder kafka_2.11-2.4.0\config (considering your version of kafka, folder name could be kafka_<kafka_version>)
2. Search for entry log.dirs
3. If your log.dir path contains windows directory path like this E:\Shyam\Software\kafka_2.11-2.4.0\kafka-logs which has a single backslash i.e , change it to double back-slash i.e with \

Solution 4

On windows version 20H2 and OS build 19042.662 escaping the log.dirs path in <kafka_dir>\config directory worked for me.

it was previously

log.dirs=C:\*********\kafka\logs

changed to

log.dirs=C:\\*********\\kafka\\logs

Might need to restart zookeeper first and then kafka after the change.

Solution 5

Try the following...

1. Enable following line in ./config/server.properties

   listeners=PLAINTEXT://:9092

2. Modify default ZooKeeper dataDir

3. Modify default Kafka log dir

View more solutions

Comments

[vanthome]

I suspect it's a problem with Kafka 2.4.0: stackoverflow.com/questions/59592518/…

[TourEiffel]

Thank you for your reply, the path of my log file in server.properties is log.dirs=C:\\Users\\Dorian\\kafka_2.12-2.4.0\\kafka-logs is it wrong ?

[TourEiffel]

I also have no data folder into kafka_2.12-2.4.0 so how should i set up the data folder path ?

[BJ5]

the logs are in /tmp directory

[Jakob]

Good for you @johnmin The file I removed still was in the logs directory.

[BJ5]

Yes, it will get generated again, but Kafka will start without issues

[Jakob]

@johnmin ok. I never claimed it wouldn't. Obviously the old file was the problem and the newly generated one worked fine. The removed file was in the logs directory.

[DarkCygnus]

hehe, second time I end up here. Thanks mate! Any advantages in choosing tmp over persistent?