Azure Storage Queue - processing messages on poison queue

Question

I've been using Azure Storage Queues to post messages too, then write the messages to a db table. However I've noticed that when an error occurs processing messages on the queue, the message is written to a poison queue.

Here is some background to the setup of my app:

Azure Web App -> Writes message to the queue

Azure function -> Queue trigger processes the message and writes the contents to a db

There was an issue with the db schema which caused the INSERTS to fail. Each message was retried 5 times, which I believe is the default for retrying queue messages, and after the 5th attempt the message was placed on the poison queue.

The db schema was subsequently fixed but now I've no way of processing the messages on the poison queue.

My question is can we recover messages written to the poison queue in order to process them and INSERT them into the db, and if so how?

Solution 1

For your particular problem, I would recommend solution mentioned in question part of this post: Azure: How to move messages from poison queue to back to main queue?

Please note that name of poison queue == $"{queueName}-poison"

Solution 2

In my current project I've created something what is called: "Support functions" in the FunctionApp. It exposes a special HTTP endpoint with Admin authorization level that can be executed at any time.

Please See the code below, which solves the problem of reprocessing messages from the poison queue:

public static class QueueOperations
{
    [FunctionName("Support_ReprocessPoisonQueueMessages")]
    public static async Task<IActionResult> Support_ReprocessPoisonQueueMessages([HttpTrigger(AuthorizationLevel.Admin, "put", Route = "support/reprocessQueueMessages/{queueName}")]HttpRequest req, ILogger log,
        [Queue("{queueName}")] CloudQueue queue,
        [Queue("{queueName}-poison")] CloudQueue poisonQueue, string queueName)
    {
        log.LogInformation("Support_ReprocessPoisonQueueMessages function processed a request.");

        int.TryParse(req.Query["messageCount"], out var messageCountParameter);
        var messageCount = messageCountParameter == 0 ? 10 : messageCountParameter;

        var processedMessages = 0;
        while (processedMessages < messageCount)
        {
            var message = await poisonQueue.GetMessageAsync();
            if (message == null)
                break;

            var messageId = message.Id;
            var popReceipt = message.PopReceipt;

            await queue.AddMessageAsync(message); // a new Id and PopReceipt is assigned
            await poisonQueue.DeleteMessageAsync(messageId, popReceipt);
            processedMessages++;
        }

        return new OkObjectResult($"Reprocessed {processedMessages} messages from the {poisonQueue.Name} queue.");
    }
}

Alternatively it may be a good idea to create a new message with the additional metadata (as information that the message has already been processed in the past with no success - then it may be send to the dead letter queue).

Comments

[Michael Freidgeim]

Does this answer your question? Azure: How to move messages from poison queue to back to main queue?

[alwayslearning]

Not sure this is the best approach in case a 'corrupted' message gets recycled forever.

[sANDwORm]

@alwayslearning Agreed, that is why I said 'for this particular problem'. That code shall run only for a short time to pump poison messages caused by the recent glitch in and then be deleted. Code could also ignore very old (by min timestamp) messages which were not caused by the glitch.

[laffaner]

Thanks, that's what I was looking for :) You mention manual intervention, and the link to the github docs also touches on this, but what type of manual steps could you take for a message whose dequeue count has been exceeded? For example, I'm using the Azure Storage Explorer and can see the message on the poison queue. However I don't seem to be able to do anything with the message. There is an option to dequeue the message, but I don't see where the message goes after this? I'm pretty new to azure functions (as you can probably guess) so trying to get a good understanding of it.

[alwayslearning]

I haven't tried it, but if you trigger another Azure function when a message is added to the poison queue and it process the msg successfully, I assume the msg will be deleted from the poison queue. In the end, isn't that what you want to achieve - process 'failed' msgs.

[laffaner]

I agree, for my particular scenario it would work to copy the message back to the original queue and process it again now that the db has been fixed. So thanks for this. However, how would this work in a production environment. There's a potential that another error could occur and as @alwayslearning says, messages could get into an eternal loop of retries. I'm curious to find out what manual steps could be used if we notify someone when a message has been added to the queue.

[laffaner]

Ok so 2 functions, one for the main queue and another for the poison queue, doing the same processing. However if it is a problem other than the one I encountered and the message is corrupt, won't it always retry corrupt messages?? Or does the dequeue count kick in on a poison queue too?

[alwayslearning]

You can add a 'dequeueCount' int parameter to either of the functions to do some processing. You can add it to the original function and after 5 retries for example let it go the poison queue by default, or send an email for manual intervention. Check the 'Manual poison message handling' topic in the link provided in the answer. Use SendGrid output binding in the function to send an email programmatically. docs.microsoft.com/en-us/azure/azure-functions/…

[laffaner]

Thanks, there's plenty there for me to go through and try out. I'll let you know how I get on :)

[alwayslearning]

Feel free to upvote and mark it as an answer if you think it helped you :)

[laffaner]

I've tried to upvote, but it doesn't show up because I'm a newbie (although apparently it does register)! I'll mark it as an answer once I get playing about with it tomorrow when I'm in the office.