Determine the Kafka-Client compatibility with kafka-broker

Question

Within the remote kafka cloud cluster, the kafka brokers will be updated to a new version (5.1) and therefore a new kafka protocol applies.

Now I should update my kafka client to being able to connect. Right now I am using following kafka relevant dependencies in my spring-boot application:

<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-parent</artifactId>
<version>1.3.8.RELEASE</version>

<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-kafka</artifactId>
<version>2.0.1.RELEASE</version>

<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<version>1.0.3.RELEASE</version>

<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
<version>1.0.3.RELEASE</version>

<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.9.0.1</version>

<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>0.9.0.1</version>

I already searched for an appropriate compatibility matrix for kafka-clients and kafka-brokers.

• Does anyone have an idea how to upgrade successfully ?
• Does anyone have a comptability matrix for this dependencies?

Solution 1

There's a link to the confluent matrix on the Spring for Apache Kafka project page (along with spring-kafka/kafka-clients compatibility).

0.9 is very, very old.

Typically, clients/brokers newer than 0.10.2.0 can talk to each other, but if records have headers, you will need a client >= 0.11.0.0.

Solution 2

From https://github.com/spring-cloud/spring-cloud-stream/wiki/Kafka-Client-Compatibility

Compatibility matrix

+--------+--------------+------------------+---------------+-------------+
| Spring |  Spring for  |      Spring      | kafka-clients |    Kafka    |
|  Cloud | Apache Kafka |    Integration   |               |    Broker   |
| Stream |              | for Apache Kafka |               |             |
+--------+--------------+------------------+---------------+-------------+
| 2.1.x  | 2.2.x        | 3.1.x            | 2.0.0,        | 2.0.0,      |
|        |              |                  | 1.1.x,        | 1.1.x,      |
|        |              |                  | 1.0.x(*)      | 1.0.x,      |
|        |              |                  |               | 0.11.0.x(*) |
+--------+--------------+------------------+---------------+-------------+
| 2.0.x  | 2.1.x        | 3.0.x            | 1.1.x,        | 1.1.x,      |
|        |              |                  | 1.0.x(*)      | 1.0.x,      |
|        |              |                  |               | 0.11.0.x(*) |
+--------+--------------+------------------+---------------+-------------+
| 1.3.x  | 1.3.x,       | 2.3.x,           | 0.11.0.x(**), | 0.10.x.x    |
|        | 1.2.x,       | 2.2.x,           | 0.10.2.x      | or higher   |
|        | 1.1.x        | 2.1.x            |               |             |
+--------+--------------+------------------+---------------+-------------+
| 1.2.x  | 1.2.x,       | 2.2.x,           | 0.10.1.x      | 0.10.x.x    |
|        | 1.1.x        | 2.1.x            |               | or higher   |
+--------+--------------+------------------+---------------+-------------+

Other sources

• https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix
• https://spring.io/projects/spring-kafka

Solution 3

Bidirectional Client Compatibility is now supported, you don't need to worry about the compatibility matrix anymore, for KIP-35 enabled clients, any version are good, KIP-35 is released from Broker protocol - 0.10.0, Java clients - 0.10.2

refer:

https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix

https://www.confluent.io/blog/upgrading-apache-kafka-clients-just-got-easier/