AWS SNS : Case of multiple subscribers

Question

I am a beginner in using AWS services. I have a requirement recently in which I wanted to send some data from service 1 to service 2 and service 3. So, what I am thinking to do is, I will push notification to SNS from service 1 and service 2 and service 3 would be subscribers to this SNS topic. So, this is the case of multiple subscribers subscribed to the same topic and both subscribers want same data.

I have some doubts regarding the basic functionality of AWS SNS. If someone could help in that, that would be really helpful.

1. Say, there are 2 notifications A, B pushed to SNS topic, so will both the subscribers get both of the notification?

2. For the same scenario, when does the 2 notifications gets deleted from SNS topic?

3. Does SNS store the notifications somewhere? Or it simply passes through the messages to subscribers?

4. What would happen if one of the subscribers fails and it is not able to get some notifications? Will it get those notifications when it would again connect to SNS topic or it would not get those?

Please provide your answers to some of the above questions. These would really help me in understanding how SNS work internally.

Thanks for help.

Solution

Messages sent to SNS topics go to all subscribers.

Publishers send messages to topics. Once a new message is published, Amazon SNS attempts to deliver that message to every endpoint that is subscribed to the topic. (emphasis added)

http://docs.aws.amazon.com/sns/latest/dg/PublishTopic.html

Messages do not actually get "deleted" from a topic (you may be thinking of SQS)... but they don't persist, either. They are published, and then they are gone.

An apparent exception to this -- retry policies -- is not really an exception. The message, in this case, has been published and conceptually is already gone from the topic, but SNS can still retry delivery to a specific target.

Everything SNS does is a push. Subscribers don't connect to SNS and ask for messages. Once a message is published, it's published. No future subscribers will ever see an old message, and no subscriber that "missed" it can come back to get it.

However... SNS fanout can send messages to multiple SQS queues. In that case, you subscribe the queues to the topic, and consume messages from the queues. Each queue gets a copy of each message, and one consumer from each queue will receive a copy whenever it goes to poll SQS.

An SQS Queue also makes a good backup temporary message retention location. If you send an SNS messages to some other kind of endpoint -- HTTPS or Lambda for example, you can also send the messages to an SQS queue, but then under normal operation, don't actually poll the queue. The messages will automatically be purged from the queue after the maximum message retention period, which defaults to 4 days, but can be configured for a max of 14 days. If anything goes wrong with the topic subscriber and messages are lost, you can go retrieve them from this backup queue, but otherwise they eventually just disappear on their own when the timeout expires. There is no limit to the number of messages that can wait, unread, in a queue.

Comments

[Chorinator]

What if multiple users share the same SQS? Will all of them receive the message or for this to happen each one must have it's own SQS registered to the topic? Thanks

[Michael - sqlbot]

@Chorinator if you're fanning out a message to multiple consumers, and each of them should get a copy of every message, then each needs its own SQS queue. A single SQS queue should deliver each message only once to one consumer, regardless of the number of consumers listening to the queue. (In rare conditions, it is possible for SQS to lose track of the fact that you already received a message, and in that case to avoid lost data it will deliver it again, so technically speaking, standard queues deliver each message "at least once," but for practical purposes this is the same as "only once.")

[Michael - sqlbot]

As noted above, "one consumer from each queue will receive a copy whenever it goes to poll SQS."

[Andy Dufresne]

@Michael-sqlbot - We have the same problem to solve. Is it possible to configure SNS to push to SQS only in event of failure in delivering the message to a subscriber? Dumping all the messages in SQS queue creates a problem for the consumer to analyze all messages in the queue when it restarts.