Why do Jumbo Frames hurt latency?

Question

Recently I've been reading about jumbo frames and I got confused in a few places. As far as I understand there is a lower bound on the Ethernet frame size between 'store and forward' points (layer 2 bridges) as frames need to be still being transmitted while it reaches the destination to enable collision detection. This limit is untouched by the jumbo frame setting.

Jumbo frames increase the upper bound of frame sizes from 1500B + headers to larger values (for example 4000B or 9000B + headers).

Larger frames allows lower overhead, etc., but there is more chance that single packet will be corrupted in transit beyond error correction capabilities. If a packet is corrupted it needs to be retransmitted (whole) adding to latency. Also the transmission of the packet takes more time as it needs to be (I believe) received fully before transferring to CPU or forwarding. However latency sensitive applications usually use UDP and custom packet sizes so they would not use jumbo frames (as long as they don't do MTU discovery) so they should not be affected by Jumbo Frames as the frames would be just shorter.

Given that I read jumbo frames hurt latency to a measurable degree I started wondering what is causing this effect?

Solution

Here are some ways that jumbo frames can hurt latency. You already knew some of them:

1. A 9kB jumbo frame is 6x as large as the largest standard Ethernet frames, which are 1500 Bytes. So at the same bit error rate, jumbo frames have a 6x higher chance of having an error, and when an error happens, the whole 6x larger frame has to be retransmitted.

2. The polynomial that was selected for the CRC32 algorithm in the Ethernet Frame Check Sequence (FCS) is tuned for frame sizes up to 1500 Bytes. It's less effective for larger frame sizes, but the jumbo frames people didn't change the polynomial. So bit errors in jumbo frames are more likely to go undetected at the MAC layer, and have to be detected later at a higher layer (likely the application layer in the case of UDP/IP), which results in even greater latency before a retransmit is requested.

3. If a latency-sensitive packet ends up being queued up behind a full-size frame for access to the medium, it takes 6x as long to get on the medium if that full-size frame is a 9kB jumbo frame rather than a standard 1500 Byte frame.

4. If a latency-sensitive protocol used jumbo frames and tried to fill them up for bandwidth efficiency's sake, the delivery of first bytes of the frame would be delayed greatly while waiting for the last bytes of the frame to be constructed, before the frame can be sent onto the wire. For perhaps a worst-case example, some high-efficiency voice codecs can use a 2kbps bitrate, so a single 9k frame could be about 36 seconds of speech. Imagine having 36 seconds of lag in a VoIP call because this. Of course, as you noted, it would be so foolish to design a latency-sensitive protocol this way, that this seems too obvious to mention. Nevertheless, it is still a way that using jumbo frames could hurt latency.

Please also note that Path MTU Discovery is an IP-layer thing, so it is not TCP-specific. So UDP-based protocols can "benefit" from Path MTU Discovery. Also note that you don't have to do PMTUD to know the MTU of the local link, so if your sending host is on a jumbo frame Ethernet, it'll have its MTU set to (up to) 9000 bytes even without doing PMTUD. Something about the way you wrote your Question made me think you might not be aware of that.

P.S. Yes, frames need to be received fully before further handling, because the FCS comes at the end and is calculated across the whole frame. Also, there is no error correction in Ethernet, only detection.

Comments

[Maciej Piechotka]

It does not quite answer (or at least I don't see it). For TCP it will hurt the latency of connection initiation (Path MTU discovery) but it should not hurt the UDP as there is no Path MTU dicovery there - packet might be either dropped or not. Also it should not hurt the latency after the Path MTU was discovered as at this point no datagram which would exceed MTU would be sent.

[Maciej Piechotka]

AD PMTUD: I thought that it's combination of DF + Fragmentation Needed (or Packet Too Big in IPv6) - TCP/IP stack being concerned with throughput tries to discover the MTU (if there is more then one hop as you wrote) but UDP/IP would either use right size of have longer time with 'classical' frames if the FN/PTB would be passed to application. Is it correct description of PMTUD (sorry - I haven't touch it for a while and some details might got blurry).