Dynamic queue creation with RabbitMQ

Question

I've been learning RabbitMQ various topologies, however, I couldn't find any reference to dynamic queue creation (aka Declare Queue) emitted from a producer. The idea would be to create queues dynamically depending on a particular event (e.g a HTTP request). The queue would be temporary with a TTL set and named after the event ID. A consumer could then, subscribe to the topic "event.*" and merge all the messages related to it.

Example:

1. HTTP POST "Create user" received
2. producer creates a queue user.ID
3. push all the subsequent messages concerning the user in his queue (e.g "Add username", "Add email" ...)
4. worker gets assigned to a random queue "user.*" and merges everything into a user account
5. queue is automatically deleted after the TTL expired

Now, is this scenario feasible with RabbitMQ ?

Solution

Essentially, what you want to do is use RabbitMQ to buffer messages waiting in a set of queues (which is what a message queuing system does by definition). :)

Assuming you know what your queues are from the consuming side, you won't have any issues. There is no constraint that a producer can't create a queue. As a caveat, when queues expire, all messages in the queue are discarded (or optionally, they can be set to go to a dead-letter queue).

What code have you tried?

Edit

Upon further clarification (from your comment) - you are looking for "wildcard consuming" vs wildcard publishing. RabbitMQ does not support such a topology at the present time (this post asks for a similar feature).

What you would need to do is periodically enumerate the queues (using the RabbitMQ API); following that, your app could decide which ones to consume from. When a queue is deleted, the consumer is automatically closed.

Special Note It should be understood that what is being asked here is an anti-pattern. The typical behavior of a system using queues is to route messages to queues based upon content. Thus, a properly-orchestrated system would have a set of workers operating on one or more statically-defined queues. Different workers may take different queues, depending upon specialization. When a series of interactions results in messages being published to the queue, the workers assigned to the queues will handle the messages in a first-come-first-served fashion (but, as this post discusses, order cannot be guaranteed with multiple consumers). The desired system behavior then emerges as a composition of workers performing various functions operating on queues.

Comments

[3XX0]

That's what I was trying to avoid, knowing in advance the queues in the consuming side. Isn't there something to consume by pattern ? say I want a worker to consume whatever queue matching log.* ?

[theMayer]

Consuming does not work like publishing. You MUST know the queue you want to consume from in RabbitMQ. It would be relatively trivial to enumerate the queues from the RabbitMQ api to determine what they are.

[theMayer]

No problem. I am curious to know why you would not just want to have all the messages dump into one queue (based on a topic routing)?

[3XX0]

well, simplicity and efficiency I guess. In this design workers are just "mergers" they take a queue, build an object merging all the message in it and switch to another one (queue being garbage collected by RMQ). This doesn't require extra steps involving sorting and dispatching by message ID. No ?

[theMayer]

Yeah, it is interesting.