Apache Kafka - linger.ms and batch.size settings
No, this does not mean that the producer will wait for the batch to become full. The producer will send half-full batches and even batches with just a single message in them.
Therefore, setting the batch size too large will not cause delays in sending messages; it will just use more memory for the batches. Setting the batch size too small will add some overhead because the producer will need to send messages more frequently.
By default (linger.ms=0), the producer will send messages as soon as there is a sender thread available to send them, even if there’s just one message in the batch.
Hope it helps.
Thanks.
Related videos on Youtube
Comments
-
Vivek Agarwal almost 2 years
In kafka producer settings, what is the expected behavior if you have linger.ms set to 0 and a non zero batch.size? How long will the producer wait for batch.size to be reached before sending the messages? Will it wait forever till the size of messages is less than specified batch size OR since linger.ms is zero, it will not do any batching and just send every request?
-
Vivek Agarwal over 5 yearsSo essentially with linger.ms=0 there will be no batching irrespective of the batch.size parameter's value?
-
dossani over 5 yearslinger.ms gives upper bound on the delay for batching. By default(linger.ms =0), there's no additional wait time. A batch size of zero will disable batching entirely.
-
Vivek Agarwal over 5 yearsOk, so let's say we have defined a batch size of 10kb, then messages can be buffered in this batch till 10kb is reached or a sender thread becomes available. If a sender thread becomes available at 5kb, it will just send these 5kb worth of messages. Otherwise at 10kb it will anyways try to send explicitly. If we had linger.ms,then producer will wait at least that amount of time even if sender thread became available earlier and batch size is not reached. Correct me if I am wrong @dossani
-
dossani over 5 yearsYes, you are right. linger.ms introduces a bit of latency.
-
Gautham Honnavara over 3 yearsLatency using linger.ms is introduced to increase the chances of batching. By introducing a small delay, we can send more messages in a batch thereby improving throughput and less latency as there are less no of request sent over to Apache Kafka
-
Lahiru Chandima about 3 yearsWhat happens if application tries to write more messages when the buffer is full, but a sender thread is still not available?
-
Hiresh about 2 yearsOfcouse this answer would help, it exactly copied and pasted from lafka definitive guide. Doesnt mean it wouldnt help