Routing messages from Amazon SNS to SQS with filtering
Solution 1
EDIT
I cannot delete an accepted answer, so see the answer below for the now correct answer since this feature has been released.
Original (now incorrect) Answer (for posterity):
No it's not possible. Would be a great feature for them to add though.
The only alternative I know is to create a topic for each routing rule and then publish to the correct topic. It's not pretty, but it accomplishes the task. If you have a lot of rules, you might need more than the 3000 topics they allow. You can request an increase in topic limit from AWS on their website by following the instructions here http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html#limits_ses_quota.
Solution 2
This is possible by using message attribute filtering in SNS. After you subscribe different SQS queues to an SNS topic, you can specify attributes to filter on by using the SNS API SetSubscriptionAttributes. This will allow messages with different attributes to get routed to the correct SQS queue.
This is also not limited to SQS queues but any subscription sources on a SNS topic. For example, a single SNS topic can publishing one set of messages to Lambda, and another set to SQS.
SDK Reference: http://docs.aws.amazon.com/sns/latest/api/API_SetSubscriptionAttributes.html
More details are given here with examples: https://aws.amazon.com/blogs/compute/simplify-pubsub-messaging-with-amazon-sns-message-filtering/
Related videos on Youtube
Comments
-
Josh Glover almost 2 years
In RabbitMQ, one can create an exchange, then bind it to multiple queues, each with a routing key. This enables messaging architectures like this:
message_x / | \ foo-msg_q bar-msg_q msg-logger_q
Clients publish messages to the
message_x
exchange, which routes only messages with routing key "foo" to thefoo-msg_q
queue, only messages with the routing key "bar" to thebar-msg_q
queue, and all messages tomsg-logger_q
queue.I'm having trouble figuring out how to do this in AWS. My first thought was to set up permissions on the individual queues to accept messages based on subject, but the only available fields for permission conditions are:
- aws:CurrentTime
- aws:EpochTime
- aws:MultiFactorAuthAge
- aws:principaltype
- aws:SecureTransport
- aws:SourceArn
- aws:SourceIp
- aws:UserAgent
- aws:userid
- aws:username
None of these seem like they can be influenced by any message I publish to the
message_x
topic.Is it possible to doing something like this when using Amazon Simple Notification Service to fan out to multiple Simple Queue Service queues, with each queue receiving a subset of messages published to the topic?
-
Josh Glover over 7 years@OlivierAlbertini That's not very helpful. If you noticed above, I mentioned that this is something you can do in RabbitMQ, so I know there are other platforms that have this functionality. I asked specifically about SNS / SQS.
-
vercinDid you find any elegant way to solve your problem ?
-
Josh Glover about 10 yearsMy solution for now is
(defn send-message [routing-key msg])
, which publishes the message to the topic first, so any subscribed queues receive it, then send to the queue named byrouting-key
. So queues subscribed to the global topic have a simulated routing key of "*". -
vercin almost 7 years@JoshGlover I did not get your solution, could you please explain it one more time If you have time.
-
Caio Cunha over 6 yearsSNS recently released this feature as Puji's answer below stackoverflow.com/a/47479409/179138