RabbitMQ / ActiveMQ or Redis for over 250,000 msg/s

Question

Eventhough redis and message queueing software are usually used for different purposes, I would like to ask pros and cons of using redis for the following use case:

• group of event collectors write incoming messages as key/value . consumers fetch and delete processed keys
• load starting from 100k msg/s and going beyond 250k in short period of time (like months) target is to achieve million msg/s
• persistency is not strictly required. it is ok to lose non-journaled messages during failure
• performance is very important (so, the number of systems required to handle load)
• messages does not have to be processed in the order they arrive

do you know such use cases where redis chosen over traditional message queueing software ? or would you consider something else ?

note: I have also seen this but did not help: Real-time application newbie - Node.JS + Redis or RabbitMQ -> client/server how?

thanks

Solution

Given your requirements I would try Redis. It will perform better than other solutions and give you much finer grained control over the persistence characteristics. Depending on the language you're using you may be able to use a sharded Redis cluster (you need Redis bindings that support consistent hashing -- not all do). This will let you scale out to the volume you indicated. I've seen 10k/sec on my laptop in some basic tests.

You'll probably want to use the list operations in Redis (LPUSH for writes, BRPOP for reads) if you want queue semantics.

I have a former client that deployed Redis in production as a message queue last spring and they've been very happy with it.