ASP.NET Multithreading Web Requests

Question

I'm building a page in my ASP.NET solution that retrieves most of its data from a 3rd Party API Service.

The page itself will need to make approximately 5 individual different calls to the API to populate all of its controls, as each Web Request to the API brings back different sets of Data.

I would like to deal with each separate Web request i make on a new thread, simultaneously so that load time is reduced.

Each request I'm making looks like this :-

WebRequest request = WebRequest.Create(requestUrl);

string response = new StreamReader(request.GetResponse().GetResponseStream()).ReadToEnd();

return new JsonSerializer().Deserialize<OccupationSearch>(new JsonTextReader(new StringReader(response)));

Firstly, should i be attempting this? i.e, is it safe? and will it enhance performance by multi-threading this out.

Secondly, what is the best way to go about doing this? There's many different ways to multi-thread systems, but which will suit this task best?

Solution 1

Multi-threading is the right choice, but I would call it doing stuff asynchronously.

Anyway, you should know that multi-threading works different in IIS.

IIS worker process will finish a request once all child threads end, and this is a big problem, because you don't want to hold a worker process for a long time but re-use it for other requests. Actually it's a thread pool.

This is why ASP.NET offers its own approach to implement asynchronity and if you use the right approach, IIS will be able to process more requests at once because asynchronous work will be executed outside IIS process model.

I would suggest you to read more about ASP.NET async:

• http://www.hanselman.com/blog/TheMagicOfUsingAsynchronousMethodsInASPNET45PlusAnImportantGotcha.aspx

• http://blogs.msdn.com/b/tmarq/archive/2010/04/14/performing-asynchronous-work-or-tasks-in-asp-net-applications.aspx

Conclusion: use asynchronous work and this will make a more efficient use of server resources, but first learn more about how to do it in the right way!

Solution 2

Multithreading is not recommended on ASP.NET; because ASP.NET/IIS does its own multithreading, and any multithreading you do will interfere with those heuristics.

What you really want is concurrency, more specifically asynchronous concurrency since your operations are I/O-bound.

The best approach is to use HttpClient with the Task-based Asynchronous Pattern:

public async Task<OccupationSearch> GetOccupationAsync(string requestUrl)
{
  // You can also reuse HttpClient instances instead of creating a new one each time
  using (var client = new HttpClient())
  {
    var response = client.GetStringAsync(requestUrl);
    return new JsonSerializer().Deserialize<OccupationSearch>(new JsonTextReader(new StringReader(response)));
  }
}

This is asynchronous, and you can easily make it concurent by using Task.WhenAll:

List<string> urls = ...;
OccupationSearch[] results = await Task.WhenAll(urls.Select(GetOccupationAsync));

Solution 3

I would argue that Multithreading (within reason) would provide benefits as the way your code is written know the calls to GetResponse().GetResponseStream() are blocking.

One of the easiest ways to improve performance is to use a Parallel.ForEach:

        var urls = new List<string>();

        var results = new ConcurrentBag<OccupationSearch>();

        Parallel.ForEach(urls, url =>
        {
            WebRequest request = WebRequest.Create(requestUrl);

            string response = new StreamReader(request.GetResponse().GetResponseStream()).ReadToEnd();

            var result = JsonSerializer().Deserialize<OccupationSearch>(new JsonTextReader(new StringReader(response)));

            results.Add(result);
        });

If you are using .NET 4.5 the new approach is to use async/await. Msdn has a pretty extensive article on this very topic here: http://msdn.microsoft.com/en-us/library/hh300224.aspx and http://msdn.microsoft.com/en-us/library/hh696703.aspx

Scott Hanselman also has a good looking Blog Post on this topic: http://www.hanselman.com/blog/TheMagicOfUsingAsynchronousMethodsInASPNET45PlusAnImportantGotcha.aspx