How to get last consumed offset for a consumer group?

Question

I have two consumers in a consumer group which have assigned same kafka topic partitions. I wish to get last read offset of say, consumer A from inside consumer B. Any Idea, how to implement this?

Solution 1

A single partition will never be assigned to two consumer instances in the same group.

You can use the below script to know the last consumed offset

sh kafka-consumer-groups.sh --bootstrap-server localhost:9092 --new-consumer --group groupname --describe

Solution 2

use below command at kafka->bin change your group id with groupId:

sh kafka-consumer-groups.sh --bootstrap-server localhost:29092 --group groupId --describe

you will get output like:

TOPIC  PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
topic1 0          0               35              35              -               -               -
topic2 0          1600            1600            0               -               -               -

Solution 3

Kafka store offsets by (consumer-group-id, topic, partition) so the first thing to note is that from Kafka point of view there is no such thing like "last read offset of consumer A". All information that you can get with the Kafka consumer API is for a given (group, topic, partition). You have two methods in consumer API that may be useful.

commited(): Get the last committed offset for the given partition (whether the commit happened by this process or another).

position(): Get the offset of the next record that will be fetched (if a record with that offset exists).

If that is not what you need, then you will have to implement something yourself. Assuming you already know how to get the last offset read from consumer A, then consumer A should store that value in some location that is available to consumer B. This location could be

• Kafka itself. For example consumer A can publish last read offset to a well know topic like ConsumerA-p0 and Consumer B can subscribe to this topic.
• Zookeeper. Again, agreeing in a well known path.
• An external database.
• More rudimentary options if both consumers share the same OS: IPC, a file in the file system, a variable in memory protected with a lock, etc.