minimal TCP MSS in Linux

Question

The TCP MSS in Linux must be at least 88 (include/net/tcp.h):

/* Minimal accepted MSS. It is (60+60+8) - (20+20). */
#define TCP_MIN_MSS             88U

My question is: where did they come up with "60 + 60 + 8" and why? I get that 20 + 20 comes from the IP header + TCP header.

EDIT: After taking a closer look at the headers, the formula looks for me like this:

(MAX_IP_HDR + MAX_TCP_HDR + MIN_IP_FRAG) - (MIN_IP_HDR + MIN_TCP_HDR)

The question still stands: why? Why does the Linux kernel use this formula, thereby prohibiting (a forced flow of) TCP segments of, say, 20 bytes? Think iperf here.

EDIT2: Here's my use case. By forcing a low MSS on socket/connection, all the packets sent by the stack will have a small size. I want to set a low MSS when working with iperf for packets/second testing. I can't get IP packets smaller than 128 bytes (Ethernet frames of 142 bytes) on the wire because of this lower limit for the MSS! I would like to get as close to an Ethernet frame size of 64 bytes as per RFC 2544. Theoretically this should be possible: 18 + 20 + 20 < 64.

Solution 1

An implementation is required to support the maximum-sized TCP and IP headers, which are 60 bytes each.

An implementation must support 576-byte datagrams, which even with maximum-headers means more than 8 bytes of data in the datagram. To send datagrams with more than 8 bytes of data, IP fragmentation must put at least 8 bytes of data in at least one of the packets that represent the fragments of the datagram. Thus an implementation must support at least 8 bytes of data in a packet.

Putting this together, an implementation must support 60+60+8 byte packets.

When we send packets that are part of a TCP stream, they have a 20-byte IP header (plus options) and a 20-byte TCP header (plus options). That leaves a minimum of (60+60+8)-(20+20) bytes remaining for data and options. Hence this is the maximum we can safely assume an implementation's TCP MSS.

Solution 2

I don't know where that number comes from, but I can tell you it's outside the spec. The minimum MTU supported for IP networks is 576 bytes, which is 512 data bytes plus up to 64 bytes for IP + TCP headers and TCP options. That value was chosen to give decently low overhead in the typical case.

My reading of bits of kernel code suggest that the value you're showing isn't arbitrary. There was an older practice to just use the raw constant 64 in place of TCP_MIN_MSS. Therefore, I assume there is some strange IP-over-Foo network the kernel developers came across that made them decide they could raise the value to what you see how.

What that nonstandard network type is, however, I cannot say.

Comments

[Admin]

How does this prohibit TCP segments of 20 bytes?

[Admin]

MSS stands for Maximum Segment Size, it's upper limit (not lower) for segment size in particular connection. TCP_MIN_MSS specifies lower bound for this limit. So, it does not prohibit in any way segments with less then 88 bytes, it just states that MSS for any connection should be >= 88 bytes.

[Admin]

Of course! Sorry for not being clear enough. Please see the latest edit.

[Admin]

Why did you let the bounty expire? David's answer clears things up to my satisfaction at least. The difference between his answer and mine is that we're talking about different minima. For what it's worth, there's a third minimum, that being 41, or 20+20+1 byte of TCP data. So the minimum packet size is contingent on the reason you are asking. I expect 68 is the right answer in the cases where the kernel uses TCP_MIN_MSS.

[Admin]

I'm still not satisfied with the answer. I still fail to see the reason for which the kernel does not let me impose an arbirary small MSS to an app. I would love to have (a constant stream of TCP-loaded) IP packets of 41 bytes, but I can't, because of the TCP_MIN_MSS. Why can't it be 1? What RFC would it break? What theoreticat/practical problem would it cause? Are you sure it's "outside the spec"? "Different minima"? There's only one minimum of interest here: the smallest MSS allowed by the kernel.

[Admin]

If you want to send out 1-byte TCP packets, disable the Nagle algorithm.

[Admin]

An implementation with a wire MSS of 41 bytes would have to support a 60+60+41 byte packet (since the TCP header can be 60 bytes and the IP header can be 60 bytes and the wire MSS doesn't include the headers), or 161 bytes. Linux would encode this as 161-(20+20)=121 bytes. 121 is greater than 88. So you can do it.

[Admin]

OK, so now you're stating that the TCP options are not part of the data whose size is upper-bounded by the MSS. That a Good Thing, we're making progress. BUT: setting an MSS of 41 still fails!!! Try running iperf -c localhost -m -M 41 and you will get WARNING: attempt to set TCP maxmimum segment size to 41 failed.

[Admin]

As the documentation says, you cannot set the advertised TCP MSS below what an implementation is required to support (60+60+8). (You can argue over whether this is a bug or a feature, but it's at least documented.)

[Admin]

The documentation explains that "TCP will also impose its minimum and maximum bounds over the value provided". TCP requires support for 60 byte headers and includes the IP requirement for 60 byte headers. Fragmentation requires support for a minimum 8 data bytes -- hence support for 60+60+8 byte packets is a TCP/IP requirement. That means if you send a packet with minimal headers, (20+20), there is room for (60+60+8)-(20+20) = 88 bytes of data. This is a consequence of the TCP and IP specifications, and there's nothing Linux can do about it.

[Michael Mrozek]

The MSS doesn't include the header though (it's just the payload), and the 60 shows up twice

[ChrisCornwall]

An implementation must be able to support a packet with a maximum-sized header, but we are not sending a maximum-sized header. The difference between the maximum and what we actually send is therefore available for data and should be added to the MSS.

[waprau]

OK, so you've explained the 8 bytes. I don't know what you mean by "TCP/IP" header. I know of an IP header and a TCP one. And, as Michael pointed out 60 shows up twice. And the RFC only discusses the "effective MSS" and not a minimal one.

[ChrisCornwall]

60 shows up twice, once for the IP header and once for the TCP header.

[waprau]

68 is just about fragmentation. "60 + 60 + 8" might get fragmented, so why care about fragmentation then? Even "68 + 20" might get fragmented. And why "must" the other side "accept" "60 + 60 + 8"? "Accept" as in "without fragmentation"? Bottom line: why am I not allowed to send "20 + 20" + 10 bytes of data?

[ChrisCornwall]

The point is to figure out the smallest packet an implementation is required to be able to accept. That would be a maximal-sized TCP header (60), a maximal-sized IP header (60), and the minimum amount of data required. That packet would be a fragment -- and the fragmentation spec requires an implementation to be able to accept at least 8 bytes of data or fragmentation is not possible.

[ChrisCornwall]

576 is the MTU for datagrams. In this case, it's the packet limits that matter, not the datagram limits because TCP packets set the DF bit.

[ChrisCornwall]

You are allowed to send 20 + 20 + 10 bytes of data, but only because the implementation must support headers larger than 20 bytes. An implementation is required to support the maximum header sizes (60+60), plus at least 8 bytes of data (or fragmentation wouldn't be possible). So an implementation must support 60+60+8, and we need 20+20 for headers. So the minimum possible allowable space for data and options is (60+60+8)-(20+20)

[i_saw_drones]

Minimum MTU defined for IP datagrams, and TCP packets are also IP datagrams.

[ChrisCornwall]

I think you're missing the point that this is the maximum minimum. This is the largest MSS that an implementation must support. If you hire a worker who must be able to handle at least eight cases, you can certainly make him handle six. And he might be able to handle fifty. But eight is the maximum minimum, the highest value that is guaranteed to work, the MCL (maximum case load) that is always safe to assume. To find the highest value that is guaranteed to work, we take all the guarantees we have and put them together. An implementation must support a 60 byte TCP header. And so on.

[waprau]

let us continue this discussion in chat

[waprau]

(Somehow I can't login into chat...) Are you now saying that the TCP_MIN_MSS is NOT the "smallest maximum" but the "largest minimum" (== "maximum minimum")?

[ChrisCornwall]

We want the MSS to be as large as possible for performance reasons. However, if we set it too high, it will break. So we set it to the largest value that we know will always be safe. We wouldn't expect it to ever be lower than this, since there would normally be no reason for it to be.

[ChrisCornwall]

Right, but this TCP limitation is for packets, not datagrams, because TCP datagrams never (normally) fragment. The only sense in which the 576-byte datagram rule matters is that it means the implementation must be able to support at least 8 bytes of data in a packet (hence the 8 in the formula). Otherwise, it would be impossible to fragment a 576-byte datagram.

[waprau]

I fully understand the reason for having a large MSS. But I don't get the reason for the lower bound. Why am I not allowed to set MSS to one, in order to have IP datagrams of 41 bytes?

[ChrisCornwall]

You can set the MSS to whatever you want. The point of this is to figure out what the MSS actually is, not to make up some value for it. If you set it to 1, you would senselessly harm performance, since an implementation is required to support an MSS of (60+60+8)-(20+20) bytes anyway. (Actually, one would be nearly fatal, since it would leave insufficient room for options.)

[waprau]

No, I can't set it to whatever I want, because of the kernel :) But I feel that we're getting somewhere with the discussion. Are you saying that the segment size comprises the TCP options as well (beside the useful TCP data)? Then 41 bytes should be OK for the MIN_MSS! However, RFC 879: "The MSS counts only data octets in the segment, it does not count the TCP header or the IP header."

[ChrisCornwall]

RFC879 is talking about the MSS announced over the connection, not the MSS value stored by an implementation.

[waprau]

What would be the point of announcing one MSS and actually using another MSS? Is there a RFC that specifies this "dual nature" of the MSS? The question about the TCP options still stands.

[ChrisCornwall]

The MSS you announce has to comply with the RFC. The MSS you use internally can be stored in whatever form is most useful to the implementation. (What question about TCP options?)

[waprau]

RFC 879 does not say anything about a minimum RFC. So... which RFC? The question was: "Are you saying that the segment size comprises the TCP options as well (beside the useful TCP data)?" I think this is NOT the case, since RFC 879 states that "the MSS counts only data octets in the segment, it does not count the TCP header or the IP header."