Unique Messages per Queue in AMQP?

Question

This is similar to this other question but with a bit of a twist: I read in the specification that the message-id for AMQP messages should be set by the application itself, so in theory I could use that to guarantee a certain degree of uniqueness, right?

My main question is now: In what scope is that message-id garantueed to be unique? For the messages currently enqueued inside a specific queue? Over all queues? Over the universe? :-)

And is this behaviour standardized? I plan to use RabbitMQ here, but it would be nice to have something not vendor specifc :-)

Thanks.

Solution 1

Another suggestion is according to the dump pipes - smart endpoints school of thought.

You could handle uniqueness in your application, using some sort of shared state.

We had the same problem when switching from Gearman to RabbitMQ. We use memcached to keep track of unique message ID's posted and consumers drop messages for which the message ID is already stored in memcache (duplicates). You could also check memcache before putting it on the queue altogether.

This frees you from using this feature in your message bus layer (so you can more easily switch between brokers, also those who do not guarantee uniqueness)

Solution 2

Message Id is application-specific only and may be not unique at all. You have to take care of uniqueness by yourself.

Comments

[Horst Gutmann]

What command do you use in Memcached there? From what I've seen of the commands available there it kind of sounds like a good entry for race-conditions. Wouldn't Redis' SETNX be more useful here?

[Joost Pastoor]

You can use add() which ensures only 1 process can set the key. (Rest gets Memcached::RES_NOTSTORED). This works for a single memcached instance, not sure how it works in a cluster.