Apache Kafka - linger.ms and batch.size settings

18,122

No, this does not mean that the producer will wait for the batch to become full. The producer will send half-full batches and even batches with just a single message in them.

Therefore, setting the batch size too large will not cause delays in sending messages; it will just use more memory for the batches. Setting the batch size too small will add some overhead because the producer will need to send messages more frequently.

By default (linger.ms=0), the producer will send messages as soon as there is a sender thread available to send them, even if there’s just one message in the batch.

Hope it helps.

Thanks.

Share:
18,122

Related videos on Youtube

Vivek Agarwal
Author by

Vivek Agarwal

computer science undergrad

Updated on June 04, 2022

Comments

  • Vivek Agarwal
    Vivek Agarwal almost 2 years

    In kafka producer settings, what is the expected behavior if you have linger.ms set to 0 and a non zero batch.size? How long will the producer wait for batch.size to be reached before sending the messages? Will it wait forever till the size of messages is less than specified batch size OR since linger.ms is zero, it will not do any batching and just send every request?

  • Vivek Agarwal
    Vivek Agarwal over 5 years
    So essentially with linger.ms=0 there will be no batching irrespective of the batch.size parameter's value?
  • dossani
    dossani over 5 years
    linger.ms gives upper bound on the delay for batching. By default(linger.ms =0), there's no additional wait time. A batch size of zero will disable batching entirely.
  • Vivek Agarwal
    Vivek Agarwal over 5 years
    Ok, so let's say we have defined a batch size of 10kb, then messages can be buffered in this batch till 10kb is reached or a sender thread becomes available. If a sender thread becomes available at 5kb, it will just send these 5kb worth of messages. Otherwise at 10kb it will anyways try to send explicitly. If we had linger.ms,then producer will wait at least that amount of time even if sender thread became available earlier and batch size is not reached. Correct me if I am wrong @dossani
  • dossani
    dossani over 5 years
    Yes, you are right. linger.ms introduces a bit of latency.
  • Gautham Honnavara
    Gautham Honnavara over 3 years
    Latency using linger.ms is introduced to increase the chances of batching. By introducing a small delay, we can send more messages in a batch thereby improving throughput and less latency as there are less no of request sent over to Apache Kafka
  • Lahiru Chandima
    Lahiru Chandima about 3 years
    What happens if application tries to write more messages when the buffer is full, but a sender thread is still not available?
  • Hiresh
    Hiresh about 2 years
    Ofcouse this answer would help, it exactly copied and pasted from lafka definitive guide. Doesnt mean it wouldnt help