TCP connection reset in linux(strange packet loss) but not on windows

Question

It is all good on windows, but on linux when I try to retrieve a specific web page, I get a long wait and then a "connection reset by peer"

Pinging destination IP works fine.

I tried to reduce interface MTU to 1476(found using "ping -c1 -M do -s") and even lower values but it did not solve the problem.

On another linux PC near the destination host, there is no problem, so i suspect some router in the path.

These are wireshark and tshark output:

Linux with connection reset: http://pastebin.com/tpjS5qZc

Windows with no problem: http://pastebin.com/iyN1GDxT

It seems that third packet gets lost in the path to destination host and destination sends back several duplicate ack packets, but i can not see any relevant difference in windows and linux packets.

Solution

In your capture both servers are setting "Do not fragment bit". This means that both ends are trying to do Path MTU Discovery.

It seems that there is a firewall that blocks ICMP Fragmentation Needed form your Linux server towards the remote server. As a workaround enable MSS clamping with:

iptables -A OUTPUT -p tcp --tcp-flags SYN,RST SYN -j TCPMSS  --clamp-mss-to-pmtu

You can also try to disable P MTU Discovery in Linux:

echo  1  |sudo tee /proc/sys/net/ipv4/ip_no_pmtu_disc

From the iptables man page:

TCPMSS This target allows to alter the MSS value of TCP SYN packets, to control the maximum size for that connection (usually limiting it to your outgoing interface's MTU minus 40 for IPv4 or 60 for IPv6, respectively). Of course, it can only be used in conjunction with -p tcp.

   This  target is used to overcome criminally braindead ISPs or servers which block "ICMP Fragmentation Needed" or "ICMPv6 Packet Too
   Big" packets.  The symptoms of this problem are that everything works fine from your Linux firewall/router, but machines behind  it
   can never exchange large packets:
    1) Web browsers connect, then hang with no data received.
    2) Small mail works fine, but large emails hang.
    3) ssh works fine, but scp hangs after initial handshaking.
   Workaround: activate this option and add a rule to your firewall configuration like:

           iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN
                       -j TCPMSS --clamp-mss-to-pmtu

   --set-mss value
          Explicitly  sets  MSS  option  to  specified  value.  If  the  MSS of the packet is already lower than value, it will not be
          increased (from Linux 2.6.25 onwards) to avoid more problems with hosts relying on a proper MSS.

   --clamp-mss-to-pmtu
          Automatically clamp MSS value to (path_MTU - 40 for IPv4; -60 for IPv6).  This may not function as desired where  asymmetric
          routes  with  differing  path MTU exist — the kernel uses the path MTU which it would use to send packets from itself to the
          source and destination IP addresses. Prior to Linux 2.6.25, only the path MTU to the destination IP address  was  considered
          by this option; subsequent kernels also consider the path MTU to the source IP address.

   These options are mutually exclusive.

See: http://lartc.org/howto/lartc.cookbook.mtu-mss.html

Edit: After I've take a closer look on the captures, I've discovered that there is a broken firewall along the path that is filtering all IP packets that use TCP Timestamp option. Just run on the Linux box: echo 0 | sudo tee /proc/sys/net/ipv4/tcp_timestamps

Comments

[Mohammad Salehe]

I did both setting no_mtu_disc and MSS clamping. The only change that I see is DF flag not being set on outgoing packets: pastebin.com/gfKaznT5

[Mohammad Salehe]

This was the same when I tested from different linuxes across the world, but there was no problem from the destination country(Iran). Could be this some kind of intentional filtering?! I have no problem with other web sites in Iran it seems.

[Mircea Vutcovici]

I think it is network misconfiguration somewhere on the path, not intentional filtering.

[Mohammad Salehe]

I had tried that, did not help. Is there some way for further inspection?

[Mircea Vutcovici]

Can you also try with a HTTP request that will return a very small page. E.g. start with a page that do not exists and should return a 404. Or a page that returns a redirection. Find a page like this with the Windows box.

[Mohammad Salehe]

Actually the page I am trying is a small redirect. ok, I'm trying to get this page: ime.co.ir

[Mircea Vutcovici]

It is a broken firewall along the path. Just run on the Linux box: echo 0 | sudo tee /proc/sys/net/ipv4/tcp_timestamps

[Mircea Vutcovici]

See also: prowiki.isc.upenn.edu/wiki/TCP_tuning_for_broken_firewalls

[Mircea Vutcovici]

Welcome ;) I will try using dirk-loss.de/scapy-doc/usage.html to figure out which router is doing this. I will try to send the GET request packet with a lower TTL.