How do I download a large file (via HTTP) in .NET?
Solution 1
If you use WebClient.DownloadFile you could save it directly into a file.
Solution 2
The WebClient class is the one for simplified scenarios. Once you get past simple scenarios (and you have), you'll have to fall back a bit and use WebRequest.
With WebRequest, you'll have access to the response stream, and you'll be able to loop over it, reading a bit and writing a bit, until you're done.
From the Microsoft documentation:
We don't recommend that you use WebRequest or its derived classes for new development. Instead, use the System.Net.Http.HttpClient class.
Source: docs.microsoft.com/WebRequest
Example:
public void MyDownloadFile(Uri url, string outputFilePath)
{
const int BUFFER_SIZE = 16 * 1024;
using (var outputFileStream = File.Create(outputFilePath, BUFFER_SIZE))
{
var req = WebRequest.Create(url);
using (var response = req.GetResponse())
{
using (var responseStream = response.GetResponseStream())
{
var buffer = new byte[BUFFER_SIZE];
int bytesRead;
do
{
bytesRead = responseStream.Read(buffer, 0, BUFFER_SIZE);
outputFileStream.Write(buffer, 0, bytesRead);
} while (bytesRead > 0);
}
}
}
}
Note that if WebClient.DownloadFile works, then I'd call it the best solution. I wrote the above before the "DownloadFile" answer was posted. I also wrote it way too early in the morning, so a grain of salt (and testing) may be required.
Solution 3
You need to get the response stream and then read in blocks, writing each block to a file to allow memory to be reused.
As you have written it, the whole response, all 2GB, needs to be in memory. Even on a 64bit system that will hit the 2GB limit for a single .NET object.
Update: easier option. Get WebClient
to do the work for you: with its DownloadFile
method which will put the data directly into a file.
Solution 4
WebClient.OpenRead returns a Stream, just use Read to loop over the contents, so the data is not buffered in memory but can be written in blocks to a file.
Solution 5
i would use something like this
Nick Cartwright
Updated on July 28, 2022Comments
-
Nick Cartwright almost 2 years
I need to download a large file (2 GB) over HTTP in a C# console application. Problem is, after about 1.2 GB, the application runs out of memory.
Here's the code I'm using:
WebClient request = new WebClient(); request.Credentials = new NetworkCredential(username, password); byte[] fileData = request.DownloadData(baseURL + fName);
As you can see... I'm reading the file directly into memory. I'm pretty sure I could solve this if I were to read the data back from HTTP in chunks and write it to a file on disk.
How could I do this?