AWS SQS + SNS + Lambda

Question

I was wondering if I can send a message to SQS queue and subscribe an SNS topic to it to trigger a lambda for sending an email.

SQS -> SNS -> (Lambda) -> SES

I know SNS messages can be sent to SQS but I'm curious if the other way around is possible

Solution 1

One thing I did was to create a CloudWatch alarm on ApproximateNumberOfMessagesVisible (>= 1 for 5 minutes) for the SQS queue. The alarm publishes to an SNS topic that triggers the lambda function. The lambda function loops until it clears the queue.

It can take up to 5 minutes to trigger from the alarm, but it works fantastically for batch-scheduled tasks without needing to poll the queue. (Alarm granularity is 5 minutes for active queues.)

Solution 2

You can't go SQS -> SNS, only SNS -> SQS.

Lambda now supports scheduling so one option is to implement an SQS poller in a Lambda function and run it frequently.

Another option to consider is whether you actually need a queue. Lambda supports asynchronous processing (via the Event invocation mode) and should transparently scale horizontally to handle parallel invocations. If your lambda function doesn't require access to a central state store which might constrain parallel execution then you could probably just run all your invocations in parallel. I believe there's a 100 concurrent execution limit per account though, so you may need to batch your messages to stay under that.

Solution 3

SQS queue can be subscribed to SNS topic and so to process received SNS messages. Currently, it is not doable in other direction without additional coding (see e.g. Lambda FAQ).

I would say there is a couple of options how to do it but it is not so elegant as using more common event-driven system AWS event->SQS->Lambda. Otherwise you may need to customize/implement the code how SQS queues are processed:

1. you can implement your own event sources
2. you can have some intermediate EC2 instance to listen to SQS queues and then to trigger Lambda on SQS events

Comments

[Brian F Leighty]

I'm thinking about doing this exact same thing. Do you know if you get charged each time cloudwatch checks your queue? Not that we're really talking big bucks with this sort of thing just more curious.

[Brian F Leighty]

Also, I'm assuming from what you say the alarm keeps sending SNS of the alarm over and over? What do you have the timeout on your lambda function set to?

[Brian F Leighty]

One other thing. Sorry for all the comments. I was a little worried with the "Approximate" part have you ever had a time where there was a message waiting there that didn't get dealt with since it thought there was only 0 items instead of the 1 item there?

[squidpickles]

@BrianFLeighty the checks are free. But it doesn't alarm continuously; it may require a second alarm for messages sitting on the queue (added between lambda execution completion and next alarm). Polling may be a better solution; mine works for a typically very quiet queue (hence not worth polling.)

[squidpickles]

Missed the "approximate" comment. That seems to work fine, but ymmv

[nik.shornikov]

there is also nothing wrong with using traditional queue poller tricks with lambda: e.g. if lambda dequeues a message, during execution, then retrigger the function at the end; otherwise let it execute next as scheduled