How to resolve "Leader not available" Kafka error when trying to consume

11,952
  1. It might be, but probably is not. My guess is the topic isn't created, so kafkacat echoes the massage on screen but doesn't really send it to kafka. All the topics are probably deleted after you delete the /tmp/kafka-logs
  2. No. I don't think this is the way to look for a solution.
  3. Having a docker container is definitely the way to go - you'll soon end up running kafka on multiple brokers, examining the replication behavior, high availability scenarios etc.. Having it dockerised helps a lot.
Share:
11,952
Jon Erickson
Author by

Jon Erickson

Software Engineer with experience developing and supporting systems across the entire stack with recent focus on back-end, scalable systems. Also, recently completed an MS in Computer Science from Georgia Tech.

Updated on June 17, 2022

Comments

  • Jon Erickson
    Jon Erickson almost 2 years

    I'm playing around with Kafka and using my own local single instance of zookeeper + kafka and running into this error that I don't seem to understand how to resolve.

    I started a simple server per the Apache Kafka Quickstart Guide

    $ bin/zookeeper-server-start.sh config/zookeeper.properties
    $ bin/kafka-server-start.sh config/server.properties
    

    Then utilizing kafkacat (installed via Homebrew) I started a Producer that will just echo messages that I type into the console

    $ kafkacat -P -b localhost:9092 -t TestTopic -T
    test1
    test1
    

    But when I try to consume those messages I get an error:

    $ kafkacat -C -b localhost:9092 -t TestTopic
    % ERROR: Topic TestTopic error: Broker: Leader not available
    

    And similarly when I try to list its' metadata

    $ kafkacat -L -b localhost:9092 -t TestTopic
    Metadata for TestTopic (from broker -1: localhost:9092/bootstrap):
     0 brokers:
     1 topics:
      topic "TestTopic" with 0 partitions: Broker: Leader not available (try again)
    

    My questions:

    1. Is this an issue with my running instance of zookeeper and/or kafkacat - I ask this because I've been constantly shutting them down and restarting them, after deleting the /tmp/zookeeper and /tmp/kafka-logs directories
    2. Is there some simple setting that I need to try? I tried adding auto.leader.rebalance.enable=true in Kafka's server.properties settings file, but that didn't fix this particular issue
    3. How do I do a fresh restart of zookeeper/kafka. Is shutting them down, deleting the /tmp/zookeeper and /tmp/kafka-logs directories and then restarting zookeeper and then kafka the way to go? (Well maybe the way to go is to build a docker container that I can stand-up and tear down, I was going to use the spotify/docker-kafka container but that is not on Kafka 0.9.0.0 and I haven't taking the time to build my own)