Deciding between HttpClient and WebClient

Question

Our web app is running in .Net Framework 4.0. The UI calls controller methods through ajax calls.

We need to consume REST service from our vendor. I am evaluating the best way to call REST service in .Net 4.0. The REST service requires Basic Authentication Scheme and it can return data in both XML and JSON. There is no requirement for uploading/downloading huge data and I don't see anything in future. I took a look at few open source code projects for REST consumption and didn't find any value in those to justify additional dependency in the project. Started to evaluate WebClient and HttpClient. I downloaded HttpClient for .Net 4.0 from NuGet.

I searched for differences between WebClient and HttpClient and this site mentioned that single HttpClient can handle concurrent calls and it can reuse resolved DNS, cookie config and authentication. I am yet to see practical values that we may gain due to the differences.

I did a quick performance test to find how WebClient (sync calls), HttpClient (sync and async) perform. and here are the results:

Using same HttpClient instance for all the requests (min - max)

WebClient sync: 8 ms - 167 ms
HttpClient sync: 3 ms - 7228 ms
HttpClient async: 985 - 10405 ms

Using a new HttpClient for each request (min - max)

WebClient sync: 4 ms - 297 ms
HttpClient sync: 3 ms - 7953 ms
HttpClient async: 1027 - 10834 ms

Code

public class AHNData
{
    public int i;
    public string str;
}

public class Program
{
    public static HttpClient httpClient = new HttpClient();
    private static readonly string _url = "http://localhost:9000/api/values/";

    public static void Main(string[] args)
    {
       #region "Trace"
       Trace.Listeners.Clear();

       TextWriterTraceListener twtl = new TextWriterTraceListener(
           "C:\\Temp\\REST_Test.txt");
       twtl.Name = "TextLogger";
       twtl.TraceOutputOptions = TraceOptions.ThreadId | TraceOptions.DateTime;

       ConsoleTraceListener ctl = new ConsoleTraceListener(false);
       ctl.TraceOutputOptions = TraceOptions.DateTime;

       Trace.Listeners.Add(twtl);
       Trace.Listeners.Add(ctl);
       Trace.AutoFlush = true;
       #endregion

       int batchSize = 1000;

       ParallelOptions parallelOptions = new ParallelOptions();
       parallelOptions.MaxDegreeOfParallelism = batchSize;

       ServicePointManager.DefaultConnectionLimit = 1000000;

       Parallel.For(0, batchSize, parallelOptions,
           j =>
           {
               Stopwatch sw1 = Stopwatch.StartNew();
               GetDataFromHttpClientAsync<List<AHNData>>(sw1);
           });
       Parallel.For(0, batchSize, parallelOptions,
            j =>
            {
                Stopwatch sw1 = Stopwatch.StartNew();
                GetDataFromHttpClientSync<List<AHNData>>(sw1);
            });
       Parallel.For(0, batchSize, parallelOptions,
            j =>
            {
                using (WebClient client = new WebClient())
                {
                   Stopwatch sw = Stopwatch.StartNew();
                   byte[] arr = client.DownloadData(_url);
                   sw.Stop();

                   Trace.WriteLine("WebClient Sync " + sw.ElapsedMilliseconds);
                }
           });

           Console.Read();
        }

        public static T GetDataFromWebClient<T>()
        {
            using (var webClient = new WebClient())
            {
                webClient.BaseAddress = _url;
                return JsonConvert.DeserializeObject<T>(
                    webClient.DownloadString(_url));
            }
        }

        public static void GetDataFromHttpClientSync<T>(Stopwatch sw)
        {
            HttpClient httpClient = new HttpClient();
            var response = httpClient.GetAsync(_url).Result;
            var obj = JsonConvert.DeserializeObject<T>(
                response.Content.ReadAsStringAsync().Result);
            sw.Stop();

            Trace.WriteLine("HttpClient Sync " + sw.ElapsedMilliseconds);
        }

        public static void GetDataFromHttpClientAsync<T>(Stopwatch sw)
        {
           HttpClient httpClient = new HttpClient();
           var response = httpClient.GetAsync(_url).ContinueWith(
              (a) => {
                 JsonConvert.DeserializeObject<T>(
                    a.Result.Content.ReadAsStringAsync().Result);
                 sw.Stop();
                 Trace.WriteLine("HttpClient Async " + sw.ElapsedMilliseconds);
              }, TaskContinuationOptions.None);
        }
    }
}

My Questions

1. The REST calls return in 3-4s which is acceptable. Calls to REST service are initiated in controller methods which gets invoked from ajax calls. To begin with, the calls run in a different thread and doesn't block UI. So, can I just stick with sync calls?
2. The above code was run in my localbox. In prod setup, DNS and proxy lookup will be involved. Is there any advantage of using HttpClient over WebClient?
3. Is HttpClient concurrency better than WebClient ? From the test results, I see WebClient sync calls perform better.
4. Will HttpClient be a better design choice if we upgrade to .Net 4.5? Performance is the key design factor.

Solution 1

HttpClient is the newer of the APIs and it has the benefits of

• has a good asynchronous programming model
• being worked on by Henrik F Nielson who is basically one of the inventors of HTTP, and he designed the API so it is easy for you to follow the HTTP standard, e.g. generating standards-compliant headers
• is in the .NET framework 4.5, so it has some guaranteed level of support for the forseeable future
• also has the xcopyable/portable-framework version of the library if you want to use it on other platforms - .NET 4.0, Windows Phone, etc.

If you are writing a web service which is making REST calls to other web services, you should want to be using an asynchronous programming model for all your REST calls, so that you don't hit thread starvation. You probably also want to use the newest C# compiler which has async/await support.

Note: It isn't more performant, AFAIK. It's probably somewhat similarly performant if you create a fair test.

Solution 2

HttpClientFactory

It's important to evaluate the different ways you can create an HttpClient, and part of that is understanding HttpClientFactory.

https://docs.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests

This is not a direct answer I know - but you're better off starting here than ending up with new HttpClient(...) everywhere.

Solution 3

When it comes to ASP.NET apps I still prefer WebClient over HttpClient because:

1. The modern implementation comes with async/awaitable task-based methods
2. Has smaller memory footprint and 2-5 times faster (other answers already mention that)
3. It's suggested to "reuse a single instance of HttpClient for the lifetime of your application". But ASP.NET has no "lifetime of application", only lifetime of a request. The current guidance for ASP.NET 5 is to use HttpClientFactory, but it can only be used via dependency injection. Some people want a simpler solution.
4. Most importantly, if you're using one singleton instance of HttpClient through the lifetime of the app like MS suggests - it has known issues. For example the DNS caching issue - HttpClient simply ignores the TTL and caches DNS "forever". There are workarounds, however. If you'd like to learn more about the issues and confusion with HttpClient just read this comment at Microsoft GitHub.

Solution 4

Firstly, I am not an authority on WebClient vs. HttpClient, specifically. Secondly, from your comments above, it seems to suggest that WebClient is synchronous only whereas HttpClient is both.

I did a quick performance test to find how WebClient (synchronous calls), HttpClient (synchronous and asynchronous) perform. And here are the results.

I see that as a huge difference when thinking for future, i.e., long running processes, responsive GUI, etc. (add to the benefit you suggest by .NET framework 4.5 - which in my actual experience is hugely faster on IIS).

Solution 5

Perhaps you could think about the problem in a different way. WebClient and HttpClient are essentially different implementations of the same thing. What I recommend is implementing the Dependency Injection pattern with an IoC Container throughout your application. You should construct a client interface with a higher level of abstraction than the low level HTTP transfer. You can write concrete classes that use both WebClient and HttpClient, and then use the IoC container to inject the implementation via config.

What this would allow you to do would be to switch between HttpClient and WebClient easily so that you are able to objectively test in the production environment.

So questions like:

Will HttpClient be a better design choice if we upgrade to .Net 4.5?

Can actually be objectively answered by switching between the two client implementations using the IoC container. Here is an example interface that you might depend on that doesn't include any details about HttpClient or WebClient.

/// <summary>
/// Dependency Injection abstraction for rest clients. 
/// </summary>
public interface IClient
{
    /// <summary>
    /// Adapter for serialization/deserialization of http body data
    /// </summary>
    ISerializationAdapter SerializationAdapter { get; }

    /// <summary>
    /// Sends a strongly typed request to the server and waits for a strongly typed response
    /// </summary>
    /// <typeparam name="TResponseBody">The expected type of the response body</typeparam>
    /// <typeparam name="TRequestBody">The type of the request body if specified</typeparam>
    /// <param name="request">The request that will be translated to a http request</param>
    /// <returns></returns>
    Task<Response<TResponseBody>> SendAsync<TResponseBody, TRequestBody>(Request<TRequestBody> request);

    /// <summary>
    /// Default headers to be sent with http requests
    /// </summary>
    IHeadersCollection DefaultRequestHeaders { get; }

    /// <summary>
    /// Default timeout for http requests
    /// </summary>
    TimeSpan Timeout { get; set; }

    /// <summary>
    /// Base Uri for the client. Any resources specified on requests will be relative to this.
    /// </summary>
    Uri BaseUri { get; set; }

    /// <summary>
    /// Name of the client
    /// </summary>
    string Name { get; }
}

public class Request<TRequestBody>
{
    #region Public Properties
    public IHeadersCollection Headers { get; }
    public Uri Resource { get; set; }
    public HttpRequestMethod HttpRequestMethod { get; set; }
    public TRequestBody Body { get; set; }
    public CancellationToken CancellationToken { get; set; }
    public string CustomHttpRequestMethod { get; set; }
    #endregion

    public Request(Uri resource,
        TRequestBody body,
        IHeadersCollection headers,
        HttpRequestMethod httpRequestMethod,
        IClient client,
        CancellationToken cancellationToken)
    {
        Body = body;
        Headers = headers;
        Resource = resource;
        HttpRequestMethod = httpRequestMethod;
        CancellationToken = cancellationToken;

        if (Headers == null) Headers = new RequestHeadersCollection();

        var defaultRequestHeaders = client?.DefaultRequestHeaders;
        if (defaultRequestHeaders == null) return;

        foreach (var kvp in defaultRequestHeaders)
        {
            Headers.Add(kvp);
        }
    }
}

public abstract class Response<TResponseBody> : Response
{
    #region Public Properties
    public virtual TResponseBody Body { get; }

    #endregion

    #region Constructors
    /// <summary>
    /// Only used for mocking or other inheritance
    /// </summary>
    protected Response() : base()
    {
    }

    protected Response(
    IHeadersCollection headersCollection,
    int statusCode,
    HttpRequestMethod httpRequestMethod,
    byte[] responseData,
    TResponseBody body,
    Uri requestUri
    ) : base(
        headersCollection,
        statusCode,
        httpRequestMethod,
        responseData,
        requestUri)
    {
        Body = body;
    }

    public static implicit operator TResponseBody(Response<TResponseBody> readResult)
    {
        return readResult.Body;
    }
    #endregion
}

public abstract class Response
{
    #region Fields
    private readonly byte[] _responseData;
    #endregion

    #region Public Properties
    public virtual int StatusCode { get; }
    public virtual IHeadersCollection Headers { get; }
    public virtual HttpRequestMethod HttpRequestMethod { get; }
    public abstract bool IsSuccess { get; }
    public virtual Uri RequestUri { get; }
    #endregion

    #region Constructor
    /// <summary>
    /// Only used for mocking or other inheritance
    /// </summary>
    protected Response()
    {
    }

    protected Response
    (
    IHeadersCollection headersCollection,
    int statusCode,
    HttpRequestMethod httpRequestMethod,
    byte[] responseData,
    Uri requestUri
    )
    {
        StatusCode = statusCode;
        Headers = headersCollection;
        HttpRequestMethod = httpRequestMethod;
        RequestUri = requestUri;
        _responseData = responseData;
    }
    #endregion

    #region Public Methods
    public virtual byte[] GetResponseData()
    {
        return _responseData;
    }
    #endregion
}

Full code

HttpClient Implementation

You can use Task.Run to make WebClient run asynchronously in its implementation.

Dependency Injection, when done well helps alleviate the problem of having to make low level decisions upfront. Ultimately, the only way to know the true answer is try both in a live environment and see which one works the best. It's quite possible that WebClient may work better for some customers, and HttpClient may work better for others. This is why abstraction is important. It means that code can quickly be swapped in, or changed with configuration without changing the fundamental design of the app.

BTW: there are numerous other reasons that you should use an abstraction instead of directly calling one of these low-level APIs. One huge one being unit-testability.

View more solutions

Comments

[Scott Chamberlain]

Your test is unfair to GetDataFromHttpClientAsync because it runs first, the other invocations get to benefit of potentially having cahed data (be it on the local machine or any transparent proxy between you and the destination) and will be faster. Also, under the right conditions var response = httpClient.GetAsync("http://localhost:9000/api/values/").Res‌​ult; can result in a deadlock due to you exhausting threadpool threads. You should never block on a activity that depends on the thread pool in ThreadPool threads , you should await instead so it returns the thread back in to the pool.

[Cory Nelson]

HttpClient with Web API Client is fantastic for a JSON/XML REST client.

[user3092913]

@Scott Chamberlain - Thanks for your reply. As all the test calls run in Parallel.Foreach, there is no guarantee which one would have run first. Also, had the first call to the service was from GetDataFromHttpClientAsync , all subsequent calls from GetDataFromHttpClientAsync should have benefited from cache and run faster. I didn't see that in the result. Rgd await, we are still using 4.0. I agree with you that HttpClient in sync fashion would lead to deadlock and I am ruling that option out of my design consideration.

[user3092913]

@CoryNelson Can you please elaborate why HttpClient with Web API Client is fantastic for a JSON/XML REST client ?

[Scott Chamberlain]

From the code you posted all of the test calls are in a Parallel.For but you still run each type of test sequentially. The way Parallel works it starts with one thread and then ramps up up to ParallelOptions.MaxDegreeOfParallelism, it does not start at the max. So if the first call on the single thread took 800ms to complete, the 999 operations after it may only take .1ms (reading from the chached result). Then the cached result is also used for the other two tests (who also start at one thread) however their first test only takes .1ms because the result was ready...

[Scott Chamberlain]

Swap the order of your best performer WebClient Sync with your worst performer HttpClient Async and please tell me what results you get.

[user3092913]

I agree about the sequential order of the Parallel.For. I changed the order of tests to 1.WebClientSync 2.HttpClientsync 3. HttpClientAsync. here's the result WebClient Sync 3ms - 79 ms HttpClientSync 3 ms - 7927ms HttpClientASync 380 ms - 1694 ms

[user3092913]

HttpClientAsync is still a slow performer because of the overhead it has to incur for creating new tasks.

[JustAndrei]

Here are few words on the difference between HttpClient and WebClient: blogs.msdn.com/b/henrikn/archive/2012/02/11/…

[Akash Kava]

You are using HttpClient wrong way, you are waiting on .Result which is why HttpClient is slower, waiting for result is not same as async call.

[crush]

Has anyone done a proper test on this to see if the performance discrepancy can be explained? Also, WebClient now seems to have async methods as well.

[Pranav Singh]

@crush There is no WebClient in .Net Core but HttpClient is.

[Theophilus]

Note the warning about HttpClient at docs.microsoft.com/en-us/dotnet/api/… concerning proper use of HttpClient. This warning is true for both .NET Framework and .NET Core.

[Theophilus]

docs.microsoft.com/en-us/dotnet/api/… recommends using HttpClient for new development instead of WebClient. This is true for both .NET Framework and .NET Core.

[Zastai]

Note: that blog link now redirects to a spam site.

[Panagiotis Kanavos]

The question is moot since at least 2018 because even .NET Framework's HttpWebRequest and by extension WebClient actually use HttpClient WebClient and HttpWebRequest are just compatibility wrappers over HttpClient for some years, with the .NET Framework version having a socket exhaustion bug that was fixed in .NET Core

[crush]

WebClient does seem to have async capabilities in the latest .NET versions. I'd like to know why it seems to be outperforming HttpClient on such a massive scale.

[Anthony Horne]

According to stackoverflow.com/a/4988325/1662973, it seems to be the same, other than the fact that one is an abstraction of the other. Maybe, it depends on how the objects are used / loaded. The minimum time does support the statement that webclient is in fact an abstraction of HttpClient, so there is a millisecond worth of overhead. The framework could be being "sneaky" in how it is really pooling or disposing of webclient.

[ed22]

If it had a way to switch proxy it would be insane

[LT Dan]

See Gabriel's comment above. In short, HttpClient is much faster if you create one instance of HttpClient and reuse it.

[Panagiotis Kanavos]

Given how .NET Old has been replaced by .NET Core, have you run your benchmarks with .NET Core? By now HttpWebRequest is a wrapper over HttpClient, so WebClient is essentially a legacy adapter for WebClient

[Panagiotis Kanavos]

only lifetime of a request. that's wrong. Using DI containers to provide singleton or scoped objects is available in the older ASP.NET stacks too, only harder to use.

[Panagiotis Kanavos]

Besides, HttpWebRequest calls HttpClient in .NET Core. Which is the only platform going forward

[Alex from Jitbit]

@PanagiotisKanavos yes, but you still do not control lifetime of the application. And average "Joe the programmer" won't bother creating static/singleton vars to cache the HttpClient anyway.

[Panagiotis Kanavos]

The lifetime of the application doesn't matter, only the injected HttpClient's - or rather, the HttpClientHandler's. Which is something easily doable for all applications. And HttpWebRequest does use a cached HttpClientHandler if available. You should rerun your benchmarks. If your results say that the wrapper over a class is faster or uses less memory than the class itself, something's wrong

[Alex from Jitbit]

Also this ".NET Old has been replaced by .NET Core" - it hasn't replaced it yet, .NET Framework is still supported and will be for another 10 years at the very least (basically as long as it's part of Windows). BUt i should've probably indicated that my answer is for .NET Framework, not Core

[Seeds]

For this example, why use an abstract as opposed to an interface? (ignoring default implementations) Is it purely for the purposes of the GetResponseData() definition? Or am I missing something here?

[Christian Findlay]

I don't understand the question

[Seeds]

I'm curious why you chose to use an Abstract here, as opposed to an interface with your "Response" objects (generic and non-generic)

[Alex from Jitbit]

@PanagiotisKanavos by the way looking at the source code for HttpWebRequest, I see that it creates a new HttpCLientHandler github.com/dotnet/corefx/blob/… Can you point me otherwise?

[Panagiotis Kanavos]

And two lines lower, that's used in an HttpClient. That's the .NET Old repo though. In .NET Core the client and handler are readonly fields since August 2019 and cached since October 2019. The curent code is private static volatile HttpClient? s_cachedHttpClient;

[Panagiotis Kanavos]

As for .NET Old still being supported, it's "supported" in the same way Silverlight was "supported" in the last 10 years. No new development, patches only and after some time, only if you pay. The link you posted shows that - there were no fixes to HttpWebRequest in the last 3 years. HttpClient is disposed when it shouldn't. And even though SendRequest is async, there's no GetResponseAsync. The APM methods are wrappers over the SendRequest task

[Panagiotis Kanavos]

Which means using HttpWebRequest in .NET Framework suffers from socket exhaustion. This shows what "support" means. Microsoft know about it but isn't fixing it. If you paid to fix the problem, you'd get a binary just for you

[Panagiotis Kanavos]

In fact, HttpWebRequests uses HttpClient even in .NET Framework, although it has bugs. That GetResponse() calls HttpClient underneath and blocks with .GetAwaiter().GetResult()

[Panagiotis Kanavos]

It's worth stressing that even HttpWebRequest in .NET Framework and by extension WebClient use HttpClient since at least 2018, so the question is essentially moot

[Panagiotis Kanavos]

WebClient uses HttpClient indirectly, because HttpWebRequest uses HttpClient internally even in .NET Framework, since at least 2018. Both WebClient and HttpWebRrequest are just obsolete compatibility wrappers at this point. WebClient does have proper async methods so it doesn't need Task.Run.

[Alex from Jitbit]

@PanagiotisKanavos agreed on the .NET Core vs Framework point, even though the product is immature (personal opinion from someone who just ported a huge app, and should probably be ignored). HUGE THANKS for the source code link. Also, we are closely monitoring thread starvation on production using dotnet-counters and I confirm that WebCLient is not starving threads in .NET Core (if used async-ly)

[Mmm]

While this is an old question, it came up on my search, so I thought I'd point out that Microsoft's documentation for WebClient in .NET 5 states, "We don't recommend that you use the WebClient class for new development. Instead, use the System.Net.Http.HttpClient class."