Connect to Kafka running in Docker

Question

I setup a single node Kafka Docker container on my local machine like it is described in the Confluent documentation (steps 2-3).

In addition, I also exposed Zookeeper's port 2181 and Kafka's port 9092 so that I'll be able to connect to them from a client running on local machine:

$ docker run -d \
    -p 2181:2181 \
    --net=confluent \
    --name=zookeeper \
    -e ZOOKEEPER_CLIENT_PORT=2181 \
    confluentinc/cp-zookeeper:4.1.0

$ docker run -d \
    --net=confluent \
    --name=kafka \
    -p 9092:9092 \
    -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
    -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 \
    -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
    confluentinc/cp-kafka:4.1.0

Problem: When I try to connect to Kafka from the host machine, the connection fails because it can't resolve address: kafka:9092.

Here is my Java code:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("client.id", "KafkaExampleProducer");
props.put("key.serializer", LongSerializer.class.getName());
props.put("value.serializer", StringSerializer.class.getName());
KafkaProducer<Long, String> producer = new KafkaProducer<>(props);
ProducerRecord<Long, String> record = new ProducerRecord<>("foo", 1L, "Test 1");
producer.send(record).get();
producer.flush();

The exception:

java.io.IOException: Can't resolve address: kafka:9092
    at org.apache.kafka.common.network.Selector.doConnect(Selector.java:235) ~[kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.common.network.Selector.connect(Selector.java:214) ~[kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:864) [kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:265) [kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.sendProducerData(Sender.java:266) [kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:238) [kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:176) [kafka-clients-2.0.0.jar:na]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
Caused by: java.nio.channels.UnresolvedAddressException: null
    at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_144]
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) ~[na:1.8.0_144]
    at org.apache.kafka.common.network.Selector.doConnect(Selector.java:233) ~[kafka-clients-2.0.0.jar:na]
    ... 7 common frames omitted

Question: How to connect to Kafka running in Docker? My code is running from host machine, not Docker.

Note: I know that I could theoretically play around with DNS setup and /etc/hosts but it is a workaround - it shouldn't be like that.

There is also similar question here, however it is based on ches/kafka image. I use confluentinc based image which is not the same.

Solution 1

Disclaimer

tl;dr - A simple port forward from the container to the host will not work, and no hosts files should be modified. What exact IP/hostname + port do you want to connect to? Make sure that value is set as advertised.listeners on the broker. Make sure that address and the servers listed as part of bootstrap.servers are actually resolvable (ping an IP/hostname, use netcat to check ports...)

Below gives answers for commonly used Kafka images, but it's all the same Apache Kafka running in a container.
You're just dependent on how it is configured. And which variables make it so.

---

The below answer uses confluentinc docker images to address the question that was asked, not wurstmeister/kafka. More specifically, the latter images are not well-maintained despite being the one of the most popular Kafka docker image.

The following sections try to aggregate all the details needed to use another image.

wurstmeister/kafka

Refer their README section on listener configuration, Also read their Connectivity wiki.

bitnami/kafka

If you want a small container, try these. The images are much smaller than the Confluent ones and are much more well maintained than wurstmeister. Refer their README for listener configuration.

debezium/kafka

Docs on it are mentioned here.

Note: advertised host and port settings are deprecated. Advertised listeners covers both. Similar to the Confluent containers, Debezium can use KAFKA_ prefixed broker settings to update its properties.

Others

spotify/kafka is deprecated and outdated.
fast-data-dev or lensesio/box are great for an all in one solution, but are bloated if you only want Kafka
Your own Dockerfile - Why? Is something incomplete with these others? Start with a pull request, not starting from scratch.

For supplemental reading, a fully-functional docker-compose, and network diagrams, see this blog by @rmoff

Answer

The Confluent quickstart (Docker) document assumes all produce and consume requests will be within the Docker network.

You could fix the problem of connecting to kafka:9092 by running your Kafka client code within its own container as that uses the Docker network bridge, but otherwise you'll need to add some more environment variables for exposing the container externally, while still having it work within the Docker network.

First add a protocol mapping of PLAINTEXT_HOST:PLAINTEXT that will map the listener protocol to a Kafka protocol

Key: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
Value: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT

Then setup two advertised listeners on different ports. (kafka here refers to the docker container name; it might also be named broker, so double check your service + hostnames). Notice the protocols match the right side values of the mappings above

Key: KAFKA_ADVERTISED_LISTENERS
Value: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092

When running the container, add -p 29092:29092 for the host port mapping

---

tl;dr

(with the above settings)

And if something still doesn't work, KAFKA_LISTENERS can be set to include <PROTOCOL>://0.0.0.0:<PORT> where both options match the advertised setting and Docker-forwarded port

Client on same machine, not in a container

Advertising localhost and the associated port will let you connect outside of the container, as you'd expect.

In other words, when running any Kafka Client outside the Docker network (including CLI tools you might have installed locally), use localhost:29092 for bootstrap servers and localhost:2181 for Zookeeper (requires Docker port forwarding)

Client on another machine

If trying to connect from an external server, you'll need to advertise the external hostname/ip of the host as well as/in place of localhost.
Simply advertising localhost with a port forward will not work because Kafka protocol will still continue to advertise the listeners you've configured.

This setup requires Docker port forwarding and router port forwarding (and firewall / security group changes) if not in the same local network, for example, your container is running in the cloud and you want to interact with it from your local machine.

Client in a container, on the same host

This is the least error-prone configuration; you can use DNS service names directly.

When running an app in the Docker network, use kafka:9092 (see advertised PLAINTEXT listener config above) for bootstrap servers and zookeeper:2181 for Zookeeper, just like any other Docker service communication (doesn't require any port forwarding)

If you use separate docker run commands, or Compose files, you need to define a shared network manually

See the example Compose file for the full Confluent stack or more minimal one for a single broker.

Related question

Connect to Kafka on host from Docker (ksqlDB)

Appendix

For anyone interested in Kubernetes deployments: https://operatorhub.io/?keyword=Kafka

Solution 2

When you first connect to a kafka node, it will give you back all the kafka node and the url where to connect. Then your application will try to connect to every kafka directly.

Issue is always what is the kafka will give you as url ? It's why there is the KAFKA_ADVERTISED_LISTENERS which will be used by kafka to tell the world how it can be accessed.

Now for your use-case, there is multiple small stuff to think about:

Let say you set plaintext://kafka:9092

• This is OK if you have an application in your docker compose that use kafka. This application will get from kafka the URL with kafka that is resolvable through the docker network.
• If you try to connect from your main system or from another container which is not in the same docker network this will fail, as the kafka name cannot be resolved.

==> To fix this, you need to have a specific DNS server like a service discovery one, but it is big trouble for small stuff. Or you set manually the kafka name to the container ip in each /etc/hosts

If you set plaintext://localhost:9092

• This will be ok on your system if you have a port mapping ( -p 9092:9092 when launching kafka)
• This will fail if you test from an application on a container (same docker network or not) (localhost is the container itself not the kafka one)

==> If you have this and wish to use a kafka client in another container, one way to fix this is to share the network for both container (same ip)

Last option : set an IP in the name: plaintext://x.y.z.a:9092 ( kafka advertised url cannot be 0.0.0.0 as stated in the doc https://kafka.apache.org/documentation/#brokerconfigs_advertised.listeners )

This will be ok for everybody... BUT how can you get the x.y.z.a name ?

The only way is to hardcode this ip when you launch the container: docker run .... --net confluent --ip 10.x.y.z .... Note that you need to adapt the ip to one valid ip in the confluent subnet.

Solution 3

before zookeeper

1. docker container run --name zookeeper -p 2181:2181 zookeeper

after kafka

2. docker container run --name kafka -p 9092:9092 -e KAFKA_ZOOKEEPER_CONNECT=192.168.8.128:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://ip_address_of_your_computer_but_not_localhost!!!:9092 -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 confluentinc/cp-kafka

in kafka consumer and producer config

@Bean
public ProducerFactory<String, String> producerFactory() {
    Map<String, Object> configProps = new HashMap<>();
    configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.8.128:9092");
    configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
    configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
    return new DefaultKafkaProducerFactory<>(configProps);
}

@Bean
public ConsumerFactory<String, String> consumerFactory() {
    Map<String, Object> props = new HashMap<>();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.8.128:9092");
    props.put(ConsumerConfig.GROUP_ID_CONFIG, "group_id");
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    return new DefaultKafkaConsumerFactory<>(props);
}

I run my project with these regulations. Good luck dude.

Comments

[Ami Hollander]

did you tried to connect to "localhost:9092"?

[Sasha Shpota]

Of course I tried, see the java code I posted.

[David Maze]

Possible duplicate of Interact with kafka docker container from outside of docker host

[Marius Waldal]

Pretty sure that this is only working between docker containers all setup with networking like this. you're essentially creating a separate network (confluent) here, and the two containers (zookeeper and kafka) can talk to each other, but you cannot access it from outside directly with localhost. i think it works if you use /etc/hosts, but i'm not sure. it wouldn't be a workaround, though, because the containers are not running on localhost. they're running on confluent network. does it work if you specify the ip address instead of localhost?

[Sasha Shpota]

Thank you (+1). The second option worked for me and probably I'll stick to it. I'll accept the answer later if none appears with a better solution (as you already mentioned we loose ability to establish connection from within the container).

[Maria Pomazkina-Karpikova]

But what if I want to use only 9092? I mean 9092 outside, not 29092

[OneCricketeer]

@Maria Then change the port mapping PLAINTEXT_HOST://localhost:9092 with -p 9092:9092. You'll likely still want a separate listener for containers inside the Docker network

[Maria Pomazkina-Karpikova]

Do you mean that in this case I need only "-p 9092:9092" and don't need "-p 9092:9092" ?

[OneCricketeer]

@Maria If you have a client on the host, you will need to expose some port. That port will depend on how the advertised listeners are set.

[Maria Pomazkina-Karpikova]

Oh man, stupid me. I mean I don't need exect "-p 29092:29092" ? I can have only -p 9092:9092" ?

[OneCricketeer]

@Maria I think I've answered that 3 times now? You can, yes, but only if you have changed to PLAINTEXT_HOST://localhost:9092, as compared to my answer

[OneCricketeer]

ip_address_of_your_computer_but_not_localhost ... Localhost works fine, if you refer my answer... And Compose would be better than docker run

[İbrahim Ersin Yavaş]

Should not be localhost. Because you have to think of your containers as an external system.That's why you should point it to the computer's ip address, not its localhost.

[OneCricketeer]

You can port forward from the container, then access from your host on localhost. Did you try the settings listed in my answer? Or read rmoff.net/2018/08/02/kafka-listeners-explained ?