Increasing TCP Window Size

Question

I have some doubts over increasing TCP Window Size in application. In my C++ software application, we are sending data packets of size around 1k from client to server using TCP/IP blocking socket. Recently I came across this concept TCP Window Size. So I tried increasing the value to 64K using setsockopt() for both SO_SNDBUF and SO_RCVBUF. After increasing this value, I get some improvements in performance for WAN connection but not in LAN connection.

As per my understanding in TCP Window Size,

Client will send data packets to server. Upon reaching this TCP Window Size, it will wait to make sure ACK received from the server for the first packet in the window size. In case of WAN connection, ACK is getting delayed from the server to the client because of latency in RTT of around 100ms. So in this case, increasing TCP Window Size compensates ACK wait time and thereby improving performance.

I want to understand how the performance improves in my application.

In my application, even though TCP Window Size (Both Send and Receive Buffer) is increased using setsockopt at socket level, we still maintain the same packet size of 1k (i.e the bytes we send from client to server in a single socket send). Also we disabled Nagle algorithm (inbuilt option to consolidate small packets into a large packet thereby avoiding frequent socket call).

My doubts are as follows:

1. Since I am using blocking socket, for each data packet send of 1k, it should block if ACK doesn't come from the server. Then how does the performance improve after improving the TCP window Size in WAN connection alone ? If I misunderstood the concept of TCP Window Size, please correct me.

2. For sending 64K of data, I believe I still need to call socket send function 64 times ( since i am sending 1k per send through blocking socket) even though I increased my TCP Window Size to 64K. Please confirm this.

3. What is the maximum limit of TCP window size with windows scaling enabled with RFC 1323 algorithm ?

I am not so good in my English. If you couldn't understand any of the above, please let me know.

Solution 1

First of all, there is a big misconception evident from your question: that the TCP window size is what is controlled by SO_SNDBUF and SO_RCVBUF. This is not true.

What is the TCP window size?

In a nutshell, the TCP window size determines how much follow-up data (packets) your network stack is willing to put on the wire before receiving acknowledgement for the earliest packet that has not been acknowledged yet.

The TCP stack has to live with and account for the fact that once a packet has been determined to be lost or mangled during transmission, every packet sent, from that one onwards, has to be re-sent since packets may only be acknowledged in order by the receiver. Therefore, allowing too many unacknowledged packets to exist at the same time consumes the connection's bandwidth speculatively: there is no guarantee that the bandwidth used will actually produce anything useful.

On the other hand, not allowing multiple unacknowledged packets at the same time would simply kill the bandwidth of connections that have a high bandwidth-delay product. Therefore, the TCP stack has to strike a balance between using up bandwidth for no benefit and not driving the pipe aggressively enough (and thus allowing some of its capacity to go unused).

The TCP window size determines where this balance is struck.

What do SO_SNDBUF and SO_RCVBUF do?

They control the amount of buffer space that the network stack has reserved for servicing your socket. These buffers serve to accumulate outgoing data that the stack has not yet been able to put on the wire and data that has been received from the wire but not yet read by your application respectively.

If one of these buffers is full you won't be able to send or receive more data until some space is freed. Note that these buffers only affect how the network stack handles data on the "near" side of the network interface (before they have been sent or after they have arrived), while the TCP window affects how the stack manages data on the "far" side of the interface (i.e. on the wire).

Answers to your questions

1. No. If that were the case then you would incur a roundtrip delay for each packet sent, which would totally destroy the bandwidth of connections with high latency.

2. Yes, but that has nothing to do with either the TCP window size or with the size of the buffers allocated to that socket.

3. According to all sources I have been able to find (example), scaling allows the window to reach a maximum size of 1GB.

Solution 2

1. Since I am using blocking socket, for each data packet send of 1k, it should block if ACK doesn't come from the server.

Wrong. Sending in TCP is asynchronous. send() just transfers the data to the socket send buffer and returns. It only blocks while the socket send buffer is full.

Then how does the performance improve after improving the TCP window Size in WAN connection alone?

Because you were wrong about it blocking until it got an ACK.

1. For sending 64K of data, I believe I still need to call socket send function 64 times

Why? You could just call it once with the 64k data buffer.

( since i am sending 1k per send through blocking socket)

Why? Or is this a repetition of your misconception under (1)?

even though I increased my TCP Window Size to 64K. Please confirm this.

No. You can send it all at once. No loop required.

What is the maximum limit of TCP window size with windows scaling enabled with RFC 1323 algorithm?

Much bigger than you will ever need.

Comments

[Philipp]

TCP can dynamically change the fragment size to match the highest max-MTU of any router on the way to optimize performance. When you don't want this to happen, you can prevent this by setting the "don't fragment" flag.

[Prabu]

Thanks for your response. So far I assumed that TCP Window Size is nothing but the combination of Send and Receive Buffer Size. I will go through these concepts and get back to you. Also please tell me how to change Tcp Window Size without changing registry in windows. Can we set TCP Window Size at socket level ? i.e if we set TCP window Size for my application, will it affect other applications using TCP/IP ?

[Waihon Yew]

@Prabu: It cannot be set at socket level, the window size applies to the connection as a whole. Also, there are major changes on how to manage the connection introduced with Vista (the whole stack was re-written).

[Waihon Yew]

@Prabu: That page reads "If the buffer is too small, the TCP window cannot fully open, and this limits performance". Note that the buffer has fixed size, while the window "opens", so they cannot be the same thing. But the buffer size does limit the maximum size of the window: packets remain in the buffer until they are acknowledged. If the buffer is full we may be willing to send more unacknowledged data (grow the window) but there is nowhere to keep them, so the window cannot ever be larger than the buffer.

[Waihon Yew]

@Prabu: But the thing is that we are assuming a single socket per interface for simplicity. In reality, the window for an interface will be limited by the total size of the buffers for the sockets on that interface (and by other things of course). But since your app does not know about the buffers of those other sockets, when discussing we assume you are the only one who is using the link.

[user207421]

@Jon It is true. The maximum dynamic window size is determined by the size of the socket receive buffer at the target and the window scaling in effect if any. The dynamic window size is adjusted downwards from this value according to the amount of space left in the socket receive buffer at any time. The relationship is very strong. (b) There is no such thing as 'the window for an interface'. It is a per-connection value.

[Waihon Yew]

@EJP: It's possible that I don't recall the details right -- let me freshen up and I 'll get back to the answer.

[user207421]

Leaving window scaling aside, the current advertised window size = the current amount of free space in the socket receive buffer.

[user207421]

@Prabu TCP Window size has nothing to do with the send buffer. The maximum TCP window size equals the socket receive buffer size at the receiver, provided the window scaling in effect can describe that: otherwise it equals the maximum that can be described in the connection's window scaling. You can increase the maximum via setsockopt() with the SO_RCVBUF option, within the limits of the window scaling. You can't change the dynamic value: TCP does that. And this is not an answer, it is an extension to your question, and should have been posted as such.