How to send multiple batch messages from a list to an SQS queue using Boto3 and Python
Solution 1
Building on the answer from @liorko and some trial and error, this seems to work and is much faster than the 1-by-1 method I was using before.
import boto3
sqsResource = boto3.resource('sqs')
def write_sqs(integerList):
queue = sqsResource.get_queue_by_name(QueueName=NAMEOFQUEUEHERE)
maxBatchSize = 10 #current maximum allowed
chunks = [integerList[x:x+maxBatchSize] for x in range(0, len(integerList), maxBatchSize)]
for chunk in chunks:
entries = []
for x in chunk:
entry = {'Id': str(x),
'MessageBody': str(x),
'MessageGroupId': 'ANYTHINGYOUWANT'}
entries.append(entry)
response = queue.send_messages(Entries=entries)
Solution 2
According to the documentation Entries
is a list of messages.
For each entry in Entries
the parameters type are detailed at the link.
import boto3
sqsResource = boto3.resource('sqs')
def write_sqs(integerList):
queue = sqsResource.get_queue_by_name(QueueName=NAMEOFQUEUEHERE)
entries = []
for i in integerList:
entry = {
'Id': 'id%s' % str(integerList[i]),
'MessageBody': str(integerList[i])
}
entries.append(entry)
response = queue.send_messages(entries)
grove80904
Updated on June 30, 2022Comments
-
grove80904 over 1 year
I have a list of integers (integerList) that I'd like to pass into an SQS queue where each message into the queue is an integer from the list.
I can do this one message at a time with the
send_message()
command, and the code for that is below.import boto3 sqsResource = boto3.resource('sqs') def write_sqs(integerList): queue = sqsResource.get_queue_by_name(QueueName=NAMEOFQUEUEHERE) for i in integerList: response = queue.send_message(MessageBody=str(i), MessageGroupId='TESTING')
However, I'd like to speed up the function and send the messages in batches. Currently, AWS SQS allows batching up to 10 messages at a time with the
send_messages()
command, but I'm not sure how to build theEntries=
attribute for the batch send. I'm breaking down the integerList into smaller lists of 10 usingchunks = [integerList[x:x+10] for x in range(0, len(integerList), 10)]
, but the next steps are unclear.