HTTP not finishing over LAN. Hardware cause?
6,309
Solution 1
That log screams, to me, of either a cabling problem or a duplex mismatch between a device and the switch. I'd consider forcing the speed / duplex on the NICs on two devices (the server and a client) to 100Base-TX half-duplex and working up from there.
Shoddy patch cables, bad NIC drivers, or a failing switch could cause these behaviors, too. It's important to try to isolate the problem by taking a systematic approach-- changing one thing at a time and testing before making further changes.
Solution 2
I was seeing similar issue when connection was initiated from a Windows 2012 Server R2 Core guest on VirtualBox 5.1 over bridged network with Centos 6 as VirtualBox host. While i am still investigating the problem, disabling ECN (https://en.wikipedia.org/wiki/Explicit_Congestion_Notification) on Windows machine solved my issue. Your trace shows that ECN is disabled so it is not the answer but I thought it might help others.
EDIT: Disabling GRO (Generic Receive Offload) also work make it work:
ethtool -K eth0 gro off
Related videos on Youtube
40 : 05
How to Set Up a New PC for Gaming and Streaming in 2020
03 : 32
Alan Walker & Ruben – Heading Home (Official Music Video)
01 : 42 : 05
How to build a PC, the last guide you'll ever need!
09 : 11
What is Ethernet?
03 : 14
DevOps & SysAdmins: HTTP not finishing over LAN. Hardware cause? (4 Solutions!!)
Author by
Hans Allis
Updated on September 18, 2022Comments
-
Hans Allis over 1 year
On a customer server running Apache 2.2 on Windows server 2012 we're noticing that from time to time, some requests to the server never finish. Using wireshark I've found a bunch of duplicate ACKs get sent to the server as soon as it starts answering the request and after a few seconds a couple of retransmissions are received from the server.
The network setup is really basic with a server and some clients connected to a switch using UTP cables.
I'm not really sure what to make of this. I'm considering asking them to try different cabling, switch and/or NIC's, but would like a second opinion on that.
1 0.000000000 192.168.1.103 -> 192.168.2.100 TCP 66 52011?81 [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1 2 0.000742000 192.168.2.100 -> 192.168.1.103 TCP 66 81?52011 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1 3 0.000782000 192.168.1.103 -> 192.168.2.100 TCP 54 52011?81 [ACK] Seq=1 Ack=1 Win=65536 Len=0 4 0.001646000 192.168.1.103 -> 192.168.2.100 HTTP 606 GET /symfony/web/app.php/legacy/mutaties/afroepen.php?afroepid=16250 HTTP/1.1 5 0.002353000 192.168.2.100 -> 192.168.1.103 TCP 60 81?52011 [ACK] Seq=1 Ack=553 Win=65536 Len=0 6 0.747171000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 7 0.747172000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 8 0.747246000 192.168.1.103 -> 192.168.2.100 TCP 54 52011?81 [ACK] Seq=553 Ack=2921 Win=65536 Len=0 9 0.747504000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 10 0.747507000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 11 0.747562000 192.168.1.103 -> 192.168.2.100 TCP 54 52011?81 [ACK] Seq=553 Ack=5841 Win=65536 Len=0 12 0.748241000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 13 0.748242000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 14 0.748243000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 15 0.748244000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 16 0.748319000 192.168.1.103 -> 192.168.2.100 TCP 66 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=8761 SRE=10221 17 0.748338000 192.168.1.103 -> 192.168.2.100 TCP 66 [TCP Dup ACK 16#1] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=8761 SRE=11681 18 0.748593000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 19 0.748594000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 20 0.748595000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 21 0.748624000 192.168.1.103 -> 192.168.2.100 TCP 66 [TCP Dup ACK 16#2] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=8761 SRE=13141 22 0.748643000 192.168.1.103 -> 192.168.2.100 TCP 66 [TCP Dup ACK 16#3] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=8761 SRE=14601 23 0.748654000 192.168.1.103 -> 192.168.2.100 TCP 66 [TCP Dup ACK 16#4] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=8761 SRE=16061 24 0.748965000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 25 0.748966000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 26 0.748967000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 27 0.748999000 192.168.1.103 -> 192.168.2.100 TCP 74 [TCP Dup ACK 16#5] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=17521 SRE=18981 SLE=8761 SRE=16061 28 0.749014000 192.168.1.103 -> 192.168.2.100 TCP 74 [TCP Dup ACK 16#6] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=17521 SRE=20441 SLE=8761 SRE=16061 29 0.749249000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 30 0.749250000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 31 0.749275000 192.168.1.103 -> 192.168.2.100 TCP 74 [TCP Dup ACK 16#7] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=17521 SRE=21901 SLE=8761 SRE=16061 32 0.749289000 192.168.1.103 -> 192.168.2.100 TCP 74 [TCP Dup ACK 16#8] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=17521 SRE=23361 SLE=8761 SRE=16061 33 0.749578000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Fast Retransmission] 81?52011 [PSH, ACK] Seq=7301 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)] 34 0.749581000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 35 0.749907000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 36 0.749909000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 37 0.749910000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Out-Of-Order] 81?52011 [PSH, ACK] Seq=16061 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)] 38 0.749939000 192.168.1.103 -> 192.168.2.100 TCP 82 [TCP Dup ACK 16#9] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=24821 SRE=26281 SLE=17521 SRE=23361 SLE=8761 SRE=16061 39 0.749958000 192.168.1.103 -> 192.168.2.100 TCP 82 [TCP Dup ACK 16#10] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=24821 SRE=27741 SLE=17521 SRE=23361 SLE=8761 SRE=16061 40 0.750240000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 41 0.750241000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 42 0.750269000 192.168.1.103 -> 192.168.2.100 TCP 82 [TCP Dup ACK 16#11] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=24821 SRE=29201 SLE=17521 SRE=23361 SLE=8761 SRE=16061 43 0.750289000 192.168.1.103 -> 192.168.2.100 TCP 82 [TCP Dup ACK 16#12] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=24821 SRE=30661 SLE=17521 SRE=23361 SLE=8761 SRE=16061 44 0.750622000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 45 0.750624000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 46 0.750649000 192.168.1.103 -> 192.168.2.100 TCP 82 [TCP Dup ACK 16#13] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=24821 SRE=32121 SLE=17521 SRE=23361 SLE=8761 SRE=16061 47 0.751016000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Out-Of-Order] 81?52011 [PSH, ACK] Seq=23361 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)] 48 0.751017000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 49 0.751059000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#14] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=33581 SRE=35041 SLE=24821 SRE=32121 SLE=17521 SRE=23361 SLE=8761 SRE=16061 50 0.751549000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 51 0.751551000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 52 0.751567000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#15] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=33581 SRE=36501 SLE=24821 SRE=32121 SLE=17521 SRE=23361 SLE=8761 SRE=16061 53 0.751582000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#16] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=33581 SRE=37961 SLE=24821 SRE=32121 SLE=17521 SRE=23361 SLE=8761 SRE=16061 54 0.752392000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 55 0.752393000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 56 0.752394000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Out-Of-Order] 81?52011 [PSH, ACK] Seq=32121 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)] 57 0.752396000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 58 0.752406000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#17] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361 SLE=8761 SRE=16061 59 0.752433000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#18] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=40881 SRE=42341 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361 60 0.753551000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 61 0.753553000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 62 0.753571000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#19] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=40881 SRE=43801 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361 63 0.753588000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#20] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=40881 SRE=45261 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361 64 0.754550000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 65 0.754553000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Out-Of-Order] 81?52011 [PSH, ACK] Seq=39421 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)] 66 0.754553000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU] 67 0.754572000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#21] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=40881 SRE=46721 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361 68 0.754610000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#22] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=40881 SRE=48181 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361 69 1.055597000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Retransmission] 81?52011 [PSH, ACK] Seq=7301 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)] 70 1.664583000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Retransmission] 81?52011 [PSH, ACK] Seq=7301 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)] 71 2.867842000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Retransmission] 81?52011 [PSH, ACK] Seq=7301 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)] 72 5.274379000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Retransmission] 81?52011 [PSH, ACK] Seq=7301 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)] 73 10.071711000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Retransmission] 81?52011 [PSH, ACK] Seq=7301 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)] 74 19.685995000 192.168.2.100 -> 192.168.1.103 TCP 60 81?52011 [RST, ACK] Seq=8761 Ack=553 Win=0 Len=0 75 19.686030000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#23] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=40881 SRE=48181 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361 76 45.752325000 192.168.1.103 -> 192.168.2.100 TCP 55 [TCP Keep-Alive] [TCP Window Full] 52011?81 [ACK] Seq=552 Ack=7301 Win=65536 Len=1 77 58.686133000 192.168.1.103 -> 192.168.2.100 TCP 54 52011?81 [FIN, ACK] Seq=553 Ack=7301 Win=65536 Len=0 78 58.985830000 192.168.1.103 -> 192.168.2.100 TCP 54 [TCP Retransmission] 52011?81 [FIN, ACK] Seq=553 Ack=7301 Win=65536 Len=0 79 59.585853000 192.168.1.103 -> 192.168.2.100 TCP 54 [TCP Retransmission] 52011?81 [FIN, ACK] Seq=553 Ack=7301 Win=65536 Len=0 80 60.785894000 192.168.1.103 -> 192.168.2.100 TCP 54 [TCP Retransmission] 52011?81 [FIN, ACK] Seq=553 Ack=7301 Win=65536 Len=0 81 63.185944000 192.168.1.103 -> 192.168.2.100 TCP 54 [TCP Retransmission] 52011?81 [FIN, ACK] Seq=553 Ack=7301 Win=65536 Len=0 82 67.988211000 192.168.1.103 -> 192.168.2.100 TCP 54 [TCP Retransmission] 52011?81 [FIN, ACK] Seq=553 Ack=7301 Win=65536 Len=0 83 77.582500000 192.168.1.103 -> 192.168.2.100 TCP 54 52011?81 [RST, ACK] Seq=554 Ack=7301 Win=0 Len=0