Kafka consumer offset max value?
Solution 1
According to this post, the offset is not reset:
We don't roll back offset at this moment. Since the offset is a long, it can last for a really long time. If you write 1TB a day, you can keep going for about 4 million days.
Plus, you can always use more partitions (each partition has its own offset).
So as Luciano said, probably not worth worrying about.
Solution 2
It seems that this is not really "handled". But, taking into account that the offset is per partition, it seems this is something we should not worry about :)
Please see http://search-hadoop.com/m/uyzND1uRn8D1sSH322/rollover/v=threaded
rodolk
I'm a software engineer and architect with more than 20 years of experience. My main areas of specialization and interest are networking protocols, distributed systems (yeap, that includes cloud), performance, availability, and scalability of web services. My main programming languages are C/C++, Java, Perl, and Python. In that order. I love Linux. I have a masters in Computer Science in the area of data networks, have some patents, and several publications in the area of networking protocols, system manageability, and web services performance and scalability.
Updated on July 15, 2022Comments
-
rodolk almost 2 years
I was googling and reading Kafka documentation but I couldn't find out the max value of a consumer offset and whether there is offset wraparound after max value. I understand offset is an Int64 value so max value is 0xFFFFFFFFFFFFFFFF. If there is wraparound, how does Kafka handle this situation?
-
Nick over 6 yearsYup. Just like Y2K :)
-
rodolk over 4 yearsI think we'll have to worry about it in the following 10 years.