Choosing buffer size for FTP and HTTP transfers

Question

How does one choose the size of a buffer (bytes I read from or write to socket) for the maximum throughput when implementing a low-level HTTP and FTP transfer? My application should transfer data with HTTP or FTP on connections varying from 130 Kbps to 3 Mbps (I know the expected speed beforehand). Sometimes it's a one way transfer, sometimes it goes in both directions. Should I stick with some average buffer size or I must vary it depending on the connection speed?

Thanks.

Solution 1

First, get some measurements.

Then, after you have a reliable performance measurement, make changes to your buffer size and plot a graph of speed vs. buffer size.

Since you know the connection speeds in advance, you should be able to get some measurements of actual speeds with different actual buffer sizes.

The OS, protocol stack and network is too complex to work out an answer from first principles. You need to measure before you do anything.

Solution 2

Choose a buffer size over 8KB. 9000 is typically the largest MTU (maximum transmission unit) size used in even the fastest networks.

When you use a buffer larger than the MTU of the connection, the operating system will break it down in to MTU sized pieces as needed, and thus anything you use over the MTU will have little effect on network performance.

However, using a large buffer will likely have other effect on performance, if you're transferring files, then using large buffers may increase the read performance, thus improving the speed of your application.

So, Usually picking a nice round number like 16KB is a good idea. Definitely don't go under 1500, as this can negatively effect network performance (causing the operating system to sometimes send small packets, which decrease performance on the network).

Comments

[wasker]

Hi, thanks for your thoughts. What if we're talking about slow networks (130 kbps) and bi-directional transfer (assume, we have symmetric connection). Won't large buffers clog the transmission in one of the directions, while benefiting the other?

[SoapBox]

That depends on the physical characteristics of the link (simplex vs. duplex). Most ethernet connections are duplex, meaning they can send and receive at the same time.

[SoapBox]

But also, usually congestion in networks is caused by the number of packets (not their size). So many small packets can cause more problems than a few big ones.

[wasker]

If this makes any difference, we're talking about cellular networks.