Why is the "don't fragment" flag set in https and ssh protocols?

Question

I've found a lot of information specifying that this is the case, however, I am really looking for the reason behind this. Why is it necessary? Is it necessary?

Solution

The DF flag instructs routers who would normally fragment the packet due to it being too large for a link's MTU (and potentially deliver it out of order due to that fragmentation) to instead drop the packet and return an ICMP Fragmentation Needed packet, allowing the sending host to account for the lower MTU on the path to the destination host. This process is called "Path MTU discovery".

It's generally better to leave PMTUD alone and let it do its work, as opposed to having your TCP stack dealing with out of order fragments. However, in some cases (mainly when the necessary ICMP is blocked) the PMTUD doesn't work, and the connection breaks.

That's when you'll want to have the DF flag unset - when PMTUD can't function properly and connection problems result.

Comments

[Aiden Thompson]

Is there a reason that encrypted traffic, ie. ssh and https traffic shouldn't be fragmented?

[voretaq7]

See also "If I catch you blocking ICMP I will hunt you down, eviscerate you, and strangle you with your own entrails"

[ravi yarlagadda]

@AidenThompson Generally, letting PMTUD do its thing instead of letting fragmentation happen is better for performance for all TCP connections, including SSH and HTTPS.

[Basur]

(I.e., it has nothing to do with encryption, and everything to do with TCP.)

[Smithers]

This is one of those things that server and network worlds will never agree upon.