Kafka check queue size

Question

I am trying to check the size of the queue in Kafka for a certain topics at regular time intervals. Although, I can't figure how to even check that metric even once. I'm completely new to Kafka so I'm not sure exactly what to do for this. I assume that it will involve creating either a producer or a consumer to interact with the queue, but I've hit a roadblock.

Solution 1

I think it is not possible at the moment. You should consider Kafka topic as a infinite data stream, so the only option you have IMO - to count consumed messages in your consumers.

You can use the kafka offset monitoring tool, which will show you log size per topic partition (you have to sum up): http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing-pieces/

Solution 2

• If you want to know how many msgs left to consume by topic and by partition : programmatically, you have to query Zookeeper if you are using the high level consumer client. Datas related to the current offset position are stored under the path /kafka/consumers. Take a look at the Kafka Offset Monitor tool. It will give you the idea of the kind of datas that are stored in ZK. This behavior will change in the next release 0.9.0 as write intensive in ZK is not an optimal use case.
• If you want to know how many msgs in total in the topic : you have to count by yourself with consumers. Or mirroring messages to an another Kafka cluster dedicated to analytic purposes (stats, count, anything).

The queue size notion in Kafka is irrelevent because it is not a queue but a log. You can consume, rewind, jump as you wish to any offset.