TCP Server sends [ACK] followed by [PSH,ACK]

Question

I am working on a high-performance TCP server, and I see the server not processing fast enough on and off when I pump high traffic using a TCP client. Upon close inspection, I see spikes in "delta time" on the TCP server. And, I see the server sending an ACK and 0.8 seconds later sending PSH,ACK for the same seqno. I am seeing this pattern multiple times in the pcap. Can experts comment on why the server is sending an ACK followed by a PSH,ACK with a delay in between?

TCP SERVER PCAP

Solution

To simplify what ACK and PSH means

• ACK will always be present, it simply informs the client what was the last received byte by the server.
• PSH tells the client/server to push the bytes to the application layer (the bytes forms a full message).

The usual scenario you are used to, is more or less the following:

1. The OS has a buffer where it stores received data from the client.
2. As soon as a packet is received, it is added to the buffer.
3. The application calls the socket receive method and takes the data out of the buffer
4. The application writes back data into the socket (response)
5. the OS sends a packet with flags PSH,ACK

Now imagine those scenarios:

• step 4 does not happen (application does not write back any data, or takes too long to write it)

  => OS acknowledge the reception with just an ACK (the packet will not have any data in it), if the application decides later on to send something, it will be sent with PSH,ACK.

• the message/data sent by the server is too big to fit in one packet:

  • the first packets will not have PSH flag, and will only have the ACK flag
  • the the last packet will have the flags PSH,ACK, to inform the end of the message.

Comments

[Ross Jacobs]

It sounds like the actual problem is I see the server not processing fast enough on and off when I pump high traffic using a TCP client. While I would tell you to add more details, stack overflow focuses on programming questions, so you should ask this expanded question on server fault or networkengineering.stackexchange.com (if you are a networking professional).

[Jim D.]

In frame 95182 client sends 1244 bytes with seq 87918542. In frame 95183 server ACKs 87918542+1244=87919786. Next in frame 95184 server sends 315 bytes, and since every segment other than the initial SYN has the ACK flag set, it fills the seq with 87919786 since it has not received any additional data from the client. The PSH flag is a hint to the client TCP stack to deliver the data immediately to the application, suggesting the 315 bytes was a single write to the socket by the server.

[user2548514]

@JimD. Thanks for the detailed analysis. It helps clear some of the questions that I had. So, if I understand it correctly, the "delta time" of 0.8 seconds between frames 95183 and 95184 would indicate that the client hasn't sent any data to the server correct? The client in this case has a busy loop to write on the SSL socket.

[Jim D.]

Note that frame 95182 is not a full frame and has PSH set, so this is likely the full payload written to the socket. The data in 95182 is ACKed in 95183. Then we see no data in the capture for 0.8 seconds and then the server replies with 513 bytes. If the client is in a busy loop writing the socket, why don't we see any more data for the 0.8 seconds? There is plenty of room in the recvq and no obvious reason from the capture why it wouldn't keep sending.

[user2548514]

@JimD. Excellent point! Would a GC pause on the server cause this issue? or do you think this would definitely indicate that the client hasn't written anything to the socket.

[Jim D.]

Application GC on the server will have no impact on the TCP stack other than it won't be processing data from the recvq, and hence the recvq can fill up. However, the advertised window is 647680 so there is plenty of room, and there is no obvious reason why the client can not send. GC at client is a possible explanation among many.

[slinkin]

Thanks for the answer! Could you clarify the next things so as not to create a similar question. The first question, Do I understand correctly that tcp-segment which contains the last part of the message always has PSH flag set? I know that there are examples when PSH flag set, but the message isn't completed by last segment. The second question, Is there is a case when the last received segment contains the part of the previous message and the bytes of the new message? Thanks in advance.

[Ayoub Kaanich]

tcp-segment which contains the last part of the message always has PSH flag set? yes

[Ayoub Kaanich]

Is there is a case when the last received segment contains the part of the previous message and the bytes of the new message? I have not seen it happen in practice.

[slinkin]

I'm not sure about the second question because of the chapter: 2.8. Data Communication (rfc793). Quote: "The data in any particular segment may be the result of a single SEND call, in whole or part, or of multiple SEND calls." Do I understand it correctly that a segment can cover the next cases: { >= 1 messages }, { >=1 messages, part-of-new-message}, { part-of-prev-msg, >=1 new messages, part-of-new-msg}.