Kafka Controller could not connect to brokers
I identified the problem, it is about the certificate creation. Refer to Confluent's documentation it says:
Ensure that common name (CN) matches exactly with the fully qualified domain name (FQDN) of the server. The client compares the CN with the DNS domain name to ensure that it is indeed connecting to the desired server, not a malicious one.
I regenerate the certificates and it worked!
small_ticket
Updated on June 08, 2022Comments
-
small_ticket almost 2 years
I've a 3 node Kafka cluster(version 0.10.1.0). I've followed the steps on kafka security documentation. Here are the related configurations of one of my Kafka servers.
listeners=SSL://myhostname:9093 security.inter.broker.protocol=SSL advertised.listeners=SSL://myhostname:9093 # In order to enable hostname verification ssl.endpoint.identification.algorithm=HTTPS ssl.client.auth=required # certificate file locations ssl.keystore.location=/location/server1.keystore.jks ssl.keystore.password=changeit ssl.key.password=changeit ssl.truststore.location=/location/server.truststore.jks ssl.truststore.password=changeit # Supported TLS versions ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
I've defined 3 different keystores for all my Kafka servers and signed them with the same CA. When I start the Kafka servers the controller log constantly logs the following warning log.
WARN [Controller-0-to-broker-2-send-thread], Controller 0's connection to broker host3:9093 (id: 2 rack: null) was unsuccessful (kafka.controller.RequestSendThread) java.io.IOException: Connection to host3:9093 (id: 2 rack: null) failed at kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:83) at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:93) at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:230) at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182) at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) WARN [Controller-0-to-broker-0-send-thread], Controller 0's connection to broker host1:9093 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread) java.io.IOException: Connection to host1:9093 (id: 0 rack: null) failed at kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:83) at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:93) at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:230) at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182) at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) WARN [Controller-0-to-broker-1-send-thread], Controller 0's connection to broker host2:9093 (id: 1 rack: null) was unsuccessful (kafka.controller.RequestSendThread) java.io.IOException: Connection to host2:9093 (id: 1 rack: null) failed at kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:83) at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:93) at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:230) at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182) at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
It seems to me more serious than a warning.
Do you have any idea what may be the problem?
Thanks in advance.