Kafka Console consumer with kerberos authentication

Question

How to consume published messages from the kafka (version 0.10) server which was kerberos authorized, for the authentication keytab file is being used.

I tried with the below command but no outputs were shown.

bin/kafka-console-consumer.sh --bootstrap-server :9092 --topic --from-beginning

Solution

Kerberos-enabled clusters can pose some tricky challenges at times. I've had to deal with some of these myself.

If the Kafka Cluster is Kerberos-enabled then you'll need to supply a jaas.conf file with the Kerberos details. Try following these steps(they worked for me):

1. Create a jaas.conf file with the following contents:

KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="<path-to-the-keytab-file>"
principal="<kafka-principal>";
};

Note: I've assumed that the Kafka principal and the associated keytab is already created. If not, you'll need to create these first.

2. Create a properties file (say "consumer.properties") with the following contents:

security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=kafka

3. Then at the terminal run the following command:

$export KAFKA_OPTS="-Djava.security.auth.login.config=<path-to-jaas.conf>"

4. Execute the Kafka-console-consumer script:

$ kafka-console-consumer --topic <topic-name> --from-beginning 
--bootstrap-server <anybroker>:9092 --consumer.config <consumer.properties>

EDIT - Steps 3 and 4 could be combined just in case there is a preference to keep these as one command in the command history.

I hope this helps.

Comments

[jambox]

i find it helpful to combine steps 3 and 4 e.g. $ KAFKA_OPTS="-Djava.security.auth.login.config=<path-to-jaas.‌​conf>" kafka-console-consumer --topic <topic-name> --from-beginning --bootstrap-server <anybroker>:9092 --consumer.config <consumer.properties> Therefore the entire command is in your terminal history (assuming you use that).

[Lalit]

@jambox - Thanks a lot for your suggestion. :) I've also used it for the same reason but for this answer I thought of keeping these 2 separate to explain the commands/steps better. But I'm adding this as an edit anyway just so people reading this are aware of it as well.