Download file progressively using TIdHttp

Question

I want to implement a simple http downloader using TIdHttp (Indy10). I found two kind of code examples from the internet. Unfortunately none of them satisfy me 100%. Here is the code and I want some advise.

---

Variant 1

var
  Buffer: TFileStream;
  HttpClient: TIdHttp;
begin
  Buffer := TFileStream.Create('somefile.exe', fmCreate or fmShareDenyWrite);
  try
    HttpClient := TIdHttp.Create(nil);
    try
      HttpClient.Get('http://somewhere.com/somefile.exe', Buffer); // wait until it is done
    finally
      HttpClient.Free;
    end;
  finally
    Buffer.Free;
  end;
end;

The code is compact and very easy to understand. The problem is that it allocates disk space when downloading begins. Another problem is that we cannot show the download progress in GUI directly, unless the code is executed in a background thread (alternatively we can bind HttpClient.OnWork event).

---

Variant 2:

const
  RECV_BUFFER_SIZE = 32768;
var
  HttpClient: TIdHttp;
  FileSize: Int64;
  Buffer: TMemoryStream;
begin
  HttpClient := TIdHttp.Create(nil);
  try
    HttpClient.Head('http://somewhere.com/somefile.exe');
    FileSize := HttpClient.Response.ContentLength;

    Buffer := TMemoryStream.Create;
    try
      while Buffer.Size < FileSize do
      begin
        HttpClient.Request.ContentRangeStart := Buffer.Size;
        if Buffer.Size + RECV_BUFFER_SIZE < FileSize then
          HttpClient.Request.ContentRangeEnd := Buffer.Size + RECV_BUFFER_SIZE - 1
        else
          HttpClient.Request.ContentRangeEnd := FileSize;

        HttpClient.Get(HttpClient.URL.URI, Buffer); // wait until it is done
        Buffer.SaveToFile('somefile.exe');
      end;
    finally
      Buffer.Free;
    end;
  finally
    HttpClient.Free;
  end;
end;

First we query the file size from the server and then we download file contents in pieces. Retrieved file contents will be save to disk when they are received completely. The potential problem is we have to send multiple GET requests to the server. I am not sure if some servers (such as megaupload) might limit the number of requests within particular time period.

---

My expectations

1. The downloader should send only one GET-request to the server.
2. The disk space must not be allocated when the download begins.

Any hints are appreciated.

Solution 1

Variant #1 is the simpliest, and is how Indy is meant to be used.

Regarding the disk allocation issue, you can derive a new class from TFileStream and override its SetSize() method to do nothing. TIdHTTP will still attempt to pre-allocate the file when appropriate, but it will not actually allocate any disk space. Writing to TFileStream will grow the file as needed.

Regarding status reporting, TIdHTTP has OnWork... events for that purpose. The AWorkCountMax parameter of the OnWorkBegin will be the actual file size if known (the response is not chunked), or 0 if not known. The AWorkCount parameter of the OnWork event will be the cumulative number of bytes that have been transferred so far. If the file size is known, you can display the total percentage by simply dividing the AWorkCount by the AWorkCountMax and multiplying by 100, otherwise just display the AWorkCount value by itself. If you want to display the speed of the transfer, you can calculate that from the difference of AWorkCount values and the time intervals between multiple OnWork events.

Try this:

type
  TNoPresizeFileStream = class(TFileStream)
  procedure
    procedure SetSize(const NewSize: Int64); override;
  end;

procedure TNoPresizeFileStream.SetSize(const NewSize: Int64);
begin
end;

.

type
  TSomeClass = class(TSomething)
  ...
    TotalBytes: In64;
    LastWorkCount: Int64;
    LastTicks: LongWord;
    procedure Download;
    procedure HttpWorkBegin(ASender: TObject; AWorkMode: TWorkMode; AWorkCountMax: Int64);
    procedure HttpWork(ASender: TObject; AWorkMode: TWorkMode; AWorkCount: Int64);
    procedure HttpWorkEnd(ASender: TObject; AWorkMode: TWorkMode);
  ...
  end;

procedure TSomeClass.Download;
var
  Buffer: TNoPresizeFileStream;
  HttpClient: TIdHttp;
begin
  Buffer := TNoPresizeFileStream.Create('somefile.exe', fmCreate or fmShareDenyWrite);
  try
    HttpClient := TIdHttp.Create(nil);
    try
      HttpClient.OnWorkBegin := HttpWorkBegin;
      HttpClient.OnWork := HttpWork;
      HttpClient.OnWorkEnd := HttpWorkEnd;

      HttpClient.Get('http://somewhere.com/somefile.exe', Buffer); // wait until it is done
    finally
      HttpClient.Free;
    end;
  finally
    Buffer.Free;
  end;
end;

procedure TSomeClass.HttpWorkBegin(ASender: TObject; AWorkMode: TWorkMode; AWorkCountMax: Int64);
begin
  if AWorkMode <> wmRead then Exit;

  // initialize the status UI as needed...
  //
  // If TIdHTTP is running in the main thread, update your UI
  // components directly as needed and then call the Form's
  // Update() method to perform a repaint, or Application.ProcessMessages()
  // to process other UI operations, like button presses (for
  // cancelling the download, for instance).
  //
  // If TIdHTTP is running in a worker thread, use the TIdNotify
  // or TIdSync class to update the UI components as needed, and
  // let the OS dispatch repaints and other messages normally...

  TotalBytes := AWorkCountMax;
  LastWorkCount := 0;
  LastTicks := Ticks;
end;

procedure TSomeClass.HttpWork(ASender: TObject; AWorkMode: TWorkMode; AWorkCount: Int64);
var
  PercentDone: Integer;
  ElapsedMS: LongWord;
  BytesTransferred: Int64;
  BytesPerSec: Int64;
begin
  if AWorkMode <> wmRead then Exit;

  ElapsedMS := GetTickDiff(LastTicks, Ticks);
  if ElapsedMS = 0 then ElapsedMS := 1; // avoid EDivByZero error

  if TotalBytes > 0 then
    PercentDone := (Double(AWorkCount) / TotalBytes) * 100.0;
  else
    PercentDone := 0.0;

  BytesTransferred := AWorkCount - LastWorkCount;

  // using just BytesTransferred and ElapsedMS, you can calculate
  // all kinds of speed stats - b/kb/mb/gm per sec/min/hr/day ...
  BytesPerSec := (Double(BytesTransferred) * 1000) / ElapsedMS;

  // update the status UI as needed...

  LastWorkCount := AWorkCount;
  LastTicks := Ticks;
end;

procedure TSomeClass.HttpWorkEnd(ASender: TObject; AWorkMode: TWorkMode);
begin
  if AWorkMode <> wmRead then Exit;

  // finalize the status UI as needed...
end;

Solution 2

Here is an example that shows how to use the components OnWork to show a progress bar:

Download a File from internet programatically with an Progress event using Delphi and Indy

You should not worry about the disk allocation. Disk space that is allocated is not actually written to, so it won't damage your disks. Be happy that it is allocated so that it is not possible that another process claims the disk space and let you run out of space!

Solution 3

Do not forget to add this for the Variant 2

 : Else HttpClient.Request.ContentRangeEnd := FileSize;

Replace

   if Buffer.Size + RECV_BUFFER_SIZE < FileSize then
  HttpClient.Request.ContentRangeEnd := Buffer.Size + RECV_BUFFER_SIZE - 1;

By

   if Buffer.Size + RECV_BUFFER_SIZE < FileSize then
  HttpClient.Request.ContentRangeEnd := Buffer.Size + RECV_BUFFER_SIZE - 1;
   Else HttpClient.Request.ContentRangeEnd := FileSize;

Comments

[LU RD]

If you want a caching TFileStream, look at David's contribution here: Buffered files (for faster disk access).

[stanleyxu2005]

Just something unrelated but I saw in your code: You use Ticks to calculate time span. It is not big deal in this example. But I suggest use TDateTime to represent StartTime and StopTime and use TTimeSpan.Subtract(StopTime, StartTime) to calculate the time span. Because MSDN says the ticks will be reset if the system is run continuously for 49.7 days. If your application runs on a server, the time duration might be calculated wrong.

[Remy Lebeau]

I intentionally chose not to use TDateTime because I did not want the code affected by possible clock changes (daylight savings, user manipulation,etc). Also, the code is timing the intervals that occur between events, which will never come close to the 49.7 day limit of LongWord. GetTickDiff() accounts for the wrap-around that occurs whenever GetTickCount() wraps back to zero, so that is not a problem.

[Tom]

@RemyLebeau Like I said yesterday I tried Indy again, this time version 10 and I feel like v10 is really much better than 9 and works nicely. I already have resuming, logging via HTTP POST and I used ideas from your code above to do some stats so yes- Indy is great, thanks man!