Using IdHTTP1.Get with Delphi and Indy 9/10 on a specific web server returns exception

Question

I've a problem receiving a favicon.ico from a specific web server using Delphi and Indy 9/10. Other servers do work fine. The problem is not with this web server, as wget command line utility gets the file correctly.

here is the output from wget:

c:\a>wget http://perforce.eigenbase.org:8080/favicon.ico
SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc
syswgetrc = c:/progra~1/wget/etc/wgetrc
--2013-01-27 00:12:39--  http://perforce.eigenbase.org:8080/favicon.ico
Resolving perforce.eigenbase.org... 72.14.190.177
Connecting to perforce.eigenbase.org|72.14.190.177|:8080... connected.
HTTP request sent, awaiting response... 200 No headers, assuming HTTP/0.9
Length: unspecified
Saving to: `favicon.ico'

    [ <=>                                   ] 2.862       --.-K/s   in 0s

2013-01-27 00:12:40 (143 MB/s) - `favicon.ico' saved [2862]

Here is my Delphi Indy 9/10 example code. It generates a "Connection Closed Gracefully" Exception, and the result is an empty string.

procedure TForm1.Button1Click(Sender: TObject);
var s: string;
begin
  s := '';
  try
    s := IdHTTP1.Get('http://perforce.eigenbase.org:8080/favicon.ico');
  except
     on E: Exception do
     begin
          {$IFDEF DEBUG}ShowMessage('get error:'+E.Message){$ENDIF};
     end;
  end;
  ShowMessage(IntToStr(Length(s)));
end;

If I try the same code with a different server, for example:

s := IdHTTP1.Get('http://www.google.com/favicon.ico');

everything works just fine.

Is there a workaround to get the http://perforce.eigenbase.org:8080/favicon.ico file using IdHTTP1.Get from the server?

Solution

The reason TIdHTTP is failing is because of this key piece of information that wget is reporting:

No headers, assuming HTTP/0.9

In an HTTP 0.9 response, the HTTP status line and headers are not present at all, only the raw file data by itself, terminated by a disconnect. wget supports that, but TIdHTTP does not (even though the official HTTP 1.0 and HTTP 1.1 specs require support for recognizing HTTP 0.9 responses) . TIdHTTP supports only HTTP 1.0 and 1.1 formatted messages, which require the use of an HTTP status line and headers. For whatever reason, this particular server is choosing to send an HTTP 0.9 response for Indy's UserAgent, but is choosing to send an HTTP 1.0 response for Internet Explorer UserAgents instead. Odd.

The short-term solution is to do what @TLama said. Setting the TIdHTTP.Request.UserAgent property to mimic Internet Explorer allows TIdHTTP.Get() to work properly:

procedure TForm1.Button1Click(Sender: TObject);
var
  icon: TMemoryStream;
begin
  icon := TMemoryStream.Create;
  try
    try
      IdHTTP1.Request.UserAgent := 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1';
      IdHTTP1.Get('http://perforce.eigenbase.org:8080/favicon.ico', icon);
    except
      on E: Exception do
      begin
        {$IFDEF DEBUG}ShowMessage('get error:'+E.Message){$ENDIF};
      end;
    end;
    ShowMessage(IntToStr(icon.Size));
  finally
    icon.Free;
  end;
end;

The long-term solution would be to update TIdHTTP to support HTTP 0.9 responses, even though they are very rare to encounter nowadays. I have opened tickets in Indy's issue trackers for that.