Kafka : How to connect kafka-console-consumer to fetch remote broker topic content?
Solution 1
I had same problem , I am using kafka 0.11 and I started producer and broker in my Ubuntu VM and consumer in Windows 10.
I had changed (and uncommented)
advertised.listeners=PLAINTEXT://your.host.name:9092
to
advertised.listeners=PLAINTEXT://<myActual IP address>:9092
example:
advertised.listeners=PLAINTEXT://192.168.150.150:9092
from file config/server.properties at server/broker side (i.e. in my case it was Ubuntu VM)
Solution 2
Had the same problem. Fixed it by setting the property 'advertised.host.name' as instructed by the comments in the kafka configuration (config/server.properties), and restarting the kafka server.
Solution 3
I have created a python script which allows you to access kafka/zookeeper via ssh tunnel to consume and produce messages from your local machine.
Compatible for Linux & Mac
Usage: kafkatunnel.py [OPTIONS] COMMAND [ARGS]...
Access kafka/zookeeper via ssh tunnel to consume and produce messages from
your local machine
Commands:
aws retrieve kafka/zookeeper ip's from AWS...
manual provide the IP's of your zookeeper/kafka
try it: https://github.com/simple-machines/kafka-tunnel
notrai
Updated on July 09, 2022Comments
-
notrai almost 2 years
I have setup a kafka zookeeper and 3 brokers on one machine on ec2 with ports 9092..9094 and am trying to consume the topic content from another machine. The ports 2181 (zk), 9092, 9093 and 9094 (servers) are open to the consumer machine. I can even do a
bin/kafka-topics.sh --describe --zookeeper 172.X.X.X:2181 --topic remotetopic
which gives meTopic:remotetopic PartitionCount:1 ReplicationFactor:3 Configs: Topic: remotetopic Partition: 0 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1 Blockquote
However when i do
bin/kafka-console-consumer.sh --zookeeper 172.X.X.X:2181 --from-beginning --topic remotetopic
I getWARN Fetching topic metadata with correlation id 0 for topics [Set(remotetopic)] from broker [id:0,host:localhost,port:9092] failed (kafka.client.ClientUtils$) java.nio.channels.ClosedChannelException
Why is the consumer trying to read from localhost? Is there any option or commandline or default file where this is read from; and I can change it?
Any help would be appreciated!
-
Mohamed Seif almost 7 years"advertised.host.name = localhost" replaced 'localhost' with my ip address..thank you
-
Ahmed over 4 yearsCan we then remotely access the logs and get the information on a Java or Python client?
-
GC001 about 3 yearsAn easier way is to use 0.0.0.0 instead of an actual ip address like 192.168.150.150
-
Ali Abbas Jaffri almost 3 yearsAccording to the (docs)[kafka.apache.org/documentation/…, it is not valid to set
0.0.0.0
as ip address