What is the best memory buffer size to allocate to download a file from Internet?

Question

What is the best memory buffer size to allocate to download a file from Internet? Some of the samples said that it should be 1K. Well, I need to know in general why is it? And also what's the difference if we download a small .PNG or a large .AVI?

Stream remoteStream;
Stream localStream;
WebResponse response;

try
{
    response = request.EndGetResponse(result);

    if (response == null)
        return;

    remoteStream = response.GetResponseStream();

    var localFile = Path.Combine(FileManager.GetFolderContent(), TaskResult.ContentItem.FileName);
    localStream = File.Create(localFile);

    var buffer = new byte[1024];
    int bytesRead;

    do
    {
        bytesRead = remoteStream.Read(buffer, 0, buffer.Length);
        localStream.Write(buffer, 0, bytesRead);
        BytesProcessed += bytesRead;
    } while (bytesRead > 0);
}

Solution 1

Use at least 4KB. It's the normal page size for Windows (i.e. the granularity at which Windows itself manages memory), which means that the .Net memory allocator doesn't need to break down a 4KB page into 1KB allocations.

Of course, using a 64KB block will be faster, but only marginally so.

Solution 2

For what it's worth, I tested reading a 1484 KB text file using progressive powers of two (sizes of 2,4,8,16...). I printed out to the console window the number of milliseconds required to read each one. Much past 8192 it didn't seem like much of a difference. Here are the results on my Windows 7 64 bit machine.

2^1 = 2 :264.0151
2^2 = 4 :193.011
2^3 = 8 :175.01
2^4 = 16 :153.0088
2^5 = 32 :139.0079
2^6 = 64 :134.0077
2^7 = 128 :132.0075
2^8 = 256 :130.0075
2^9 = 512 :133.0076
2^10 = 1024 :133.0076
2^11 = 2048 :90.0051
2^12 = 4096 :69.0039
2^13 = 8192 :60.0035
2^14 = 16384 :56.0032
2^15 = 32768 :53.003
2^16 = 65536 :53.003
2^17 = 131072 :52.003
2^18 = 262144 :53.003
2^19 = 524288 :54.0031
2^20 = 1048576 :55.0031
2^21 = 2097152 :54.0031
2^22 = 4194304 :54.0031
2^23 = 8388608 :54.003
2^24 = 16777216 :55.0032

Solution 3

I have problem with remote machine closing connection when used 64K buffer when downloading from iis.

I solved the problem raising the buffer to 2M

Solution 4

It will depend on the hardware and scope too. I work for cloud deployed workloads, in server world you may find 40G Ethernet cards and you can assume MTUs of 9000 bytes. Additionally, you dont want your ethernet card interrupt your processor for every single frame. So, ignoring the middle actors in the Windows/Linux kernel you should go for a one or two folds higher: 100 * 9000 ~~ 900kB so I generally choose 512KB as default value (as long as I know this value is not oversizing the regular expected file size being downloaded)

Solution 5

2k, 4k or 8k are good choices. It is not important how much is the page size, the change in speed would be really marginal and unpredictable.

First of all, C# memory can be moved, C# uses a compacting generational garbage collector. There is not any kind of information on where data will be allocated.

Second, arrays in C# can be formed by non-contiguous area of memory! Arrays are stored contiguously in virtual memory but contiguous virtual memory doesn't mean contiguous physical memory.

Third, array data structure in C# occupies some bytes more than the content itself (it stores array size and other informations). If you allocate page size amount of bytes, using the array will switch page almost always!

I would think that optimizing code using page size can be an non-optimization.

Usually C# arrays performs very well but if you really need precise allocation of data you need to use pinned arrays or Marshal allocation, but this will slow down the garbage collector.

Using marshal allocation and unsafe code can be a little faster but really it don't worth the effort.

I would say it is better to just use your arrays without thinking too much about the page size. Use 2k, 4k or 8k buffers.

View more solutions

Comments

[musefan]

1024 - Big enough to be useful, Small enough to allow frequent progress updates (if desired)

[NoWar]

Are there any official Microsoft recommendations or best practices? Do you know?

[MSalters]

Nah, it's really not that critical. Windows will use its own buffers to store the content that's sent to you while you are still busy writing the previous chunk. That's why your buffer isn't too critical.

[NoWar]

Thanks for the point! It's correct I found some info here en.wikipedia.org/wiki/Page_(computer_memory) and here jimmoyle.com/wordpress/wp-content/uploads/downloads/2011/05/‌​…. May be I need to get the system memory page size on fly... Thanks a lot!

[Jon Hanna]

Why assume the GC is smart enough to make use of non-contiguous areas of memory and yet dumb enough not to avoid doing so when possible?

[Will Calderwood]

Interesting results. I'd say it looks more like 16k is the magic number rather than 8k. There's still a 7% improvement there.

[Kikiwa]

OS, text file, but it's also depending of your hardware and your network bandwidth so these results shouldn't be applied blindly in other projects. I'd avoid "16k is the magic number" thoughts as it's wrong by design.