Web API concurrency and scalability

Question

We are faced with the task to convert a REST service based on custom code to Web API. The service has a substantial amount of requests and operates on data that could take some time to load, but once loaded it can be cached and used to serve all of the incoming requests. The previous version of the service would have one thread responsible for loading the data and getting it into the cache. To prevent the IIS from running out of worker threads clients would get a "come back later" response until the cache was ready.

My understanding of Web API is that it has an asynchronous behavior built in by operating on tasks, and as a result the number of requests will not directly relate to the number of physical threads being held.

In the new implementation of the service I am planning to let the requests wait until the cache is ready and then make a valid reply. I have made a very rough sketch of the code to illustrate:

public class ContactsController : ApiController
{
    private readonly IContactRepository _contactRepository;

    public ContactsController(IContactRepository contactRepository)
    {
        if (contactRepository == null) 
            throw new ArgumentNullException("contactRepository");
        _contactRepository = contactRepository;
    }

    public IEnumerable<Contact> Get()
    {
        return _contactRepository.Get();
    }
}

public class ContactRepository : IContactRepository
{
    private readonly Lazy<IEnumerable<Contact>> _contactsLazy;

    public ContactRepository()
    {
        _contactsLazy = new Lazy<IEnumerable<Contact>>(LoadFromDatabase, 
            LazyThreadSafetyMode.ExecutionAndPublication);
    }

    public IEnumerable<Contact> Get()
    {
        return _contactsLazy.Value;
    }

    private IEnumerable<Contact> LoadFromDatabase()
    {
        // This method could be take a long time to execute.
        throw new NotImplementedException();
    }
}

Please do not put too much value in the design of the code - it is only constructed to illustrate the problem and is not how we did it in the actual solution. IContactRepository is registered in the IoC container as a singleton and is injected into the controller. The Lazy with LazyThreadSafetyMode.ExecutionAndPublication ensures only the first thread/request is running the initialization code, the following rquests are blocked until the initialization completes.

Would Web API be able to handle 1000 requests waiting for the initialization to complete while other requests not hitting this Lazy are being service and without the IIS running out of worker threads?

Solution

Returning Task<T> from the action will allow the code to run on the background thread (ThreadPool) and release the IIS thread. So in this case, I would change

public IEnumerable<Contact> Get()

to

public Task<IEnumerable<Contact>> Get()

Remember to return a started task otherwise the thread will just sit and do nothing.

Lazy implementation while can be useful, has got little to do with the behaviour of the Web API. So I am not gonna comment on that. With or without lazy, task based return type is the way to go for long running operations.

I have got two blog posts on this which are probably useful to you: here and here.