TCP connection stuck in SYN_RECV state despite ACK received, Linux 2.6.18, embedded, ARM

Question

My client cannot connect to my protocol port (TCP) after some network glitches, even though all other protocols (telnet/HTTP/FTP) work fine.

netstat shows that my server is listening and tcpdump on the server shows all 3 packets are exchanged:

18:29:16.578964 IP 10.9.59.10.3355 > 10.9.43.131.5084: S 2602965897:2602965897(0) win 65535 <mss 1460,nop,nop,sackOK>

18:29:16.579107 IP 10.9.43.131.5084 > 10.9.59.10.3355: S 3464857909:3464857909(0) ack 2602965898 win 5840 <mss 1460,nop,nop,sackOK>

18:29:16.579284 IP 10.9.59.10.3355 > 10.9.43.131.5084: . ack 1 win 65535

But somehow netstat -t shows the connection still in SYN_RECV, as if the ack is not seen by the TCP state machine. I have to restart my server to get it to work.

syncookie is not enabled, and I know from client code behavior and tcpdump that there is no SYN flooding.

Help much appreciated.

Solution 1

The connection is in SYN_RECV state because the kernel has received a SYN packet for a port that is in LISTENING mode, but the other end did not reply with ACK.

Check if the ACK is received by the server by running the capture on the server. Is the capture taken on the client or on the server?

Solution 2

This can happen if the listener has set the DEFER_ACCEPT option on the socket and is not yet ready to accept the connection.

Comments

[David Schwartz]

Repeat the process logging Ethernet hardware address. See if the ARP table changed. Also, doublecheck the netmask.

[poige]

@DavidSchwartz, «-1» — There's packet exchange already, according to tcpdump.

[poige]

@DavidSchwartz, and «all other protocols (telnet/HTTP/FTP) work fine».

[David Schwartz]

@poige: You're assuming that we're not in the realm of the unusual or strange. If you read the question carefully, it's quite clear that we are in that realm. Your reasoning is based on the assumption that everything is working the way it is supposed to. It's much more likely that something is not working the way ti is supposed to.

[poige]

@DavidSchwartz, Well, it's your domain then, alright. :)

[Admin]

I had the same problem. The SYN-ACK was getting lost. It was problem of incorrect routing at the receiver.

[Gambit Support]

Same problem. The arp table was corrupted. Unknown cause, but on deleting the incorrect arp entry it works.

[ivanleoncz]

Also, a considerable number of SYN-RECV, indicates a possible SYN Flood Attack performed against the host. For more info: cloudflare.com/learning/ddos/syn-flood-ddos-attack