In Zookeeper client how to look up offset for a group id / topic?

Question

We are using Kafka 2.10 and zookeeper 3.4. Following the info on http://kafka.apache.org/08/documentation.html#distributionimpl I am trying to look up the offset for a group id, topic partition, but find the offset to be empty: ls /consumers/test6/offsets/ViewerLogs/0 ---> returns []

Any suggestion how to access that value?

Thanks

Solution 1

To get the offset value, run command get /consumers/test6/offsets/ViewerLogs/0 in zookeeper

Solution 2

Take a look at this document:

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper

So, the offset will be stored in Zookeeper at:

/[clusterBasePath]/consumers/[groupId]/offsets/[topic]/[partitionId] -> long (offset)

If that znode is empty, it might mean that your consumer group hasn't owned the partitions yet (decided on a mapping from partitions to consumers).

To see the current partition owners in the consumer group, take a look at:

/[clusterBasePath]/consumers/[groupId]/owners/[topic]/[partitionId] -> string (consumerId)

From personal experience, the most common cause of this is that your your consumer group is having trouble owning partitions due to timeouts. On your consumer config, you might want to try increasing rebalance.max.retries to something like 50 (or higher) and rebalance.backoff.ms to something like 5000. Also, check your Zookeeper session timeouts and increase those if necessary.

Depending on what consumer you're using (are you using consumer groups at all?), there's also a chance that you're just not committing your offsets to Zookeeper (this is ok if you don't care too much about fault tolerance). In that case, you won't be able to find the offsets in Zookeeper and will need to get them from your consumer directly.

Comments

[FZF]

Thx so much for the info. I am using high level consumers with groupIds. At this point I am not adding a consumer to an existing group. I am just trying to validate the behavior of zookeeper when starting HL consumer with a new group id. You suggested that there is a chance that I am not commiting offsets to zzokeeper. Do I have to code explicitly for the commiting of the offsets to the zookeeper in High Level Consumers? Appreciate your help.

[Jon Bringhurst]

@FZF, the high level consumer should either be committing to Zookeeper, or, in newer versions of Kafka, it may be committing to the __consumer_offsets topic instead (which replaces Zookeeper for the offset storage in newer Kafka versions).