Kafka throws java.nio.channels.ClosedChannelException

Question

When i try to consume messages from the kafka server which is hosted in ec2 with kafka console tool (V 0.9.0.1 , i think this uses old consumer APIs) I get following exception. How can i overcome this?

#./kafka-console-consumer.sh --zookeeper zookeeper1.xx.com:2181 --topic MY_TOPIC --from-beginning

[2016-04-06 14:34:58,219] WARN Fetching topic metadata with correlation id 0 for topics [Set(MY_TOPIC)] from broker [BrokerEndPoint(1014,kafka3.xx.com,9092)] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
    at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
    at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75)
    at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:74)
    at kafka.producer.SyncProducer.send(SyncProducer.scala:119)
    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
    at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2016-04-06 14:34:58,222] WARN Fetching topic metadata with correlation id 0 for topics [Set(MY_TOPIC)] from broker [BrokerEndPoint(1013,kafka22.xx.com,9092)] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
    at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
    at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75)
    at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:74)
    at kafka.producer.SyncProducer.send(SyncProducer.scala:119)
    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
    at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2016-

Solution 1

The reason for the original closed channel exception is, i had some DNS issue, which i have solved by editing my local hosts file

Solution 2

I was able to solve this issue by setting advertised.host.name to the config file

Comments

[Ratha]

I don't think for all topics it gives same type issue pls have a look stackoverflow.com/questions/36442668/…

[Ratha]

I run into several issues with new consumer, it goes infinite loop without polling messages. So, choking with older consumer APIs(the consumer console tools with the kafka distribution) and now stated to get this type closed channel issues

[Ratha]

Here are some issues i face stackoverflow.com/questions/36393867/… stackoverflow.com/questions/36439818/…

[ponkin]

@Ratha Ok. If you running kafka on EC2 there may be an issue with private IP's - it can be resolved by setting advertised.host.name=<your external EC2 hostname>, try to search "kafka on EC2 advertised.host.name" or on stackoverflow.com/questions/30880811/…

[Ratha]

The reason for the original closed channel exception is, i had some DNS issue, which i have solved by editing my local hosts file. And the EC2 case, yes i believe my settings are right

[Ratha]

Check this thread at user kafka list //Is there any behavioural change to connect local server and remote server?//

[Abu Shumon]

Thanks a lot for hint. I had same problem for a while both in my centos and windows 10. As soon as I fixed ´hosts´ file in both machines error disappeared!