Azure Functions - run long operation in another thread

Question

I am trying to implement files conversion using Azure Functions solution. The conversion can take a lot of time. Therefore I don't want waiting for the response on the calling server. I wrote the function that returns response immediately (to indicate that service is available and converting is started) and runs conversion in separate thread. Callback URL is used to send converting result.

public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, Stream srcBlob, Binder binder, TraceWriter log)
{
    log.Info($"C# HTTP trigger function processed a request. RequestUri={req.RequestUri}");

    // Get request model
    var input = await req.Content.ReadAsAsync<ConvertInputModel>();

    //Run convert in separate thread
    Task.Run( async () => {
        //Read input blob -> convert -> upload output blob
        var convertResult = await ConvertAndUploadFile(input, srcBlob, binder, log);

        //return result using HttpClient
        SendCallback(convertResult, input.CallbackUrl); 
    });

    //Return response immediately
    return req.CreateResponse(HttpStatusCode.OK);
}

The problem that the new task breaks binding. I get exception while accessing params. So how can I run long-time operation in separate tread? Or such solution is totally wrong?

Solution 1

This pattern is not recommended (or supported) in Azure Functions. Particularly when running in the consumption plan, since the runtime won't be able to accurately manage your function's lifetime and will eventually shutdown your service.

One of the recommended (and widely used) patterns here would be to queue up this work to be processed by another function, listening on that queue, and return the response to the client right away.

With this approach, you accomplish essentially the same thing, where the actual processing will be done asynchronously, but in a reliable and efficient way (benefiting from automatic scaling to properly handle increased loads, if needed)

Do keep in mind that, when using the consumption plan, there's a function timeout of 5 minutes. If the processing is expected to take longer, you'd need to run your function on a dedicated plan with AlwaysOn enabled.

Solution 2

Your solution of running the background work inside the Azure Function is wrong like you suspected. You need a 2nd service that is designed to run these long running tasks. Here is documentation to Micosoft's best practices on azure for doing background jobs.

Comments

[Adam Levitt]

Is there some documentation on this that I could be pointed to?

[Jordan Simba]

You can also increase the execution time to 10 minutes. docs.microsoft.com/en-us/azure/azure-functions/…

[Niels Brinch]

What if my thread just takes 1 second - does the same rules still apply or could this be supported? I want to respond immediately and not make the caller wait for 1 second. (and I don't want to wait up to 10 seconds for a queue trigger to react)