How to send multiple batch messages from a list to an SQS queue using Boto3 and Python

13,346

Solution 1

Building on the answer from @liorko and some trial and error, this seems to work and is much faster than the 1-by-1 method I was using before.

import boto3

sqsResource = boto3.resource('sqs')

def write_sqs(integerList):
    queue = sqsResource.get_queue_by_name(QueueName=NAMEOFQUEUEHERE)
    maxBatchSize = 10 #current maximum allowed
    chunks = [integerList[x:x+maxBatchSize] for x in range(0, len(integerList), maxBatchSize)]
    for chunk in chunks:
        entries = []
        for x in chunk:
            entry = {'Id': str(x), 
                     'MessageBody': str(x), 
                     'MessageGroupId': 'ANYTHINGYOUWANT'}
            entries.append(entry)
        response = queue.send_messages(Entries=entries)

Solution 2

According to the documentation Entries is a list of messages.
For each entry in Entries the parameters type are detailed at the link.

import boto3

sqsResource = boto3.resource('sqs')

def write_sqs(integerList):
    queue = sqsResource.get_queue_by_name(QueueName=NAMEOFQUEUEHERE)
    entries = []

    for i in integerList:
        entry =  {
            'Id': 'id%s' % str(integerList[i]),
            'MessageBody': str(integerList[i])
            }
        entries.append(entry)

    response = queue.send_messages(entries)
Share:
13,346
grove80904
Author by

grove80904

Updated on June 30, 2022

Comments

  • grove80904
    grove80904 over 1 year

    I have a list of integers (integerList) that I'd like to pass into an SQS queue where each message into the queue is an integer from the list.

    I can do this one message at a time with the send_message() command, and the code for that is below.

    import boto3
    
    sqsResource = boto3.resource('sqs')
    
    def write_sqs(integerList):
        queue = sqsResource.get_queue_by_name(QueueName=NAMEOFQUEUEHERE)
        for i in integerList:
            response = queue.send_message(MessageBody=str(i),
                                          MessageGroupId='TESTING')
    
    

    However, I'd like to speed up the function and send the messages in batches. Currently, AWS SQS allows batching up to 10 messages at a time with the send_messages() command, but I'm not sure how to build the Entries= attribute for the batch send. I'm breaking down the integerList into smaller lists of 10 using chunks = [integerList[x:x+10] for x in range(0, len(integerList), 10)], but the next steps are unclear.