Failed to resolve 'kafka:9092': Name or service not known - docker / php-rdkafka
Solution 1
The brokers will advertise themselve using advertised.listeners (which seems to be abstracted with KAFKA_ADVERTISED_HOST_NAME in that docker image) and the clients will consequently try to connect to these advertised hosts and ports.
You will thus need to make sure the client can resolve and reach this advertised hostnames, e.g. by adding "kafka" to /etc/hosts on the client host.
Solution 2
if containers' host IP is 192.168.1.110
,could set it in docker container env:
KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://192.168.1.110:{{ kafka_port }}"
Related videos on Youtube
Comments
-
Horse almost 2 years
I am trying to get php connecting to kafka all within a docker container.
kafka php lib - https://github.com/arnaud-lb/php-rdkafka/
kafka docker container - https://hub.docker.com/r/wurstmeister/kafka/
Everything is building and running ok, but when I try to connect with the PHP producer, I get the following:
httpd_1 | %3|1490816385.542|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve 'kafka:9092': Name or service not known httpd_1 | %3|1490816385.543|ERROR|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve 'kafka:9092': Name or service not known httpd_1 | %3|1490816385.543|ERROR|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: 1/1 brokers are down
I am using the following in PHP
$rk = new RdKafka\Producer(); $rk->setLogLevel(LOG_DEBUG); $rk->addBrokers("kafka"); $topicConf = new RdKafka\TopicConf(); $topicConf->set("message.timeout.ms", 1000); $topic = $rk->newTopic("DEV", $topicConf); $topic->produce(RD_KAFKA_PARTITION_UA, 0, "Message"); $rk->poll(1000); $kafkaConf = new RdKafka\Conf(); $kafkaConf->setErrorCb(function ($rk, $err, $reason) { printf("Kafka error: %s (reason: %s)\n", rd_kafka_err2str($err), $reason); }); $kafkaConf->setDrMsgCb(function ($rk, $message) { if ($message->err) { print_r($message); } else { print_r("ok"); } });
I've played around, trying to set the host IP in both the docker-compose.yml and the PHP code, but no joy. I also had
connection refused
but I don't know if thats better or worse?If it helps, my docker-compose.yml
httpd: build: . ports: - 8180:80 volumes: - ~/www:/var/www/html zookeeper: image: wurstmeister/zookeeper ports: - "2181:2181" kafka: build: ~/kafka-docker/. links: - zookeeper ports: - "9092:9092" environment: KAFKA_ADVERTISED_HOST_NAME: kafka KAFKA_ADVERTISED_PORT: 9092 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 volumes: - /var/run/docker.sock:/var/run/docker.sock
PHP version: 7.0.17
librdkafka version: 0.9.5.0
php-rdkafka version: 2.0.1
My question: Any idea how I can connect to kafka successfully from php?
-
Horse about 7 yearsThanks for the response! Do you mean the httpd container needs to be able to resolve
kafka
, or my outer host machine? Docker usually resolves all the service names in a docker-compose internally as hostnames. This works for example when connecting to a mysql container from php, with the mysql container service name. I will have a play and see. -
Edenhill about 7 yearsThe kafka client will need to be able to resolve and reach the brokers, both the bootstrap brokers (as configured in the client) as well as the brokers' advertised hostnames. If your PHP client is triggered from httpd then the httpd container/host needs to be able to resolve & reach
-
Horse about 7 yearsThanks for the advice. I ended up using another docker container (flozano/kafka if anyone is interested) in the end, and then used the host IP in the yml file, but used the yml service name, eg
kafka
in the PHP as the broker hostname. -
darkdefender27 about 6 years@Horse Did using flozano/kafka work for you? or did you follow the approach suggested by Edenhill?
-
Horse about 6 years@Edenhill I used the hostname in the docker-compose iirc. Although in the end didn't use kafka long term.
-
Joel B almost 3 yearsThis should be the accepted answer as needing to manually adding entries to a
hosts
/resolve.conf
file isn't a portable solution. -
NickL about 2 yearsAnother option is to use
0.0.0.0
as the IP address (idea taken from dotnet-testcontainers)