Difference between Redis and Kafka

Question

Redis can be used as realtime pub-sub just as Kafka.

I am confused which one to use when.

Any use case would be a great help.

Solution 1

Redis pub-sub is mostly like a fire and forget system where all the messages you produced will be delivered to all the consumers at once and the data is kept nowhere. You have limitation in memory with respect to Redis. Also, the number of producers and consumers can affect the performance in Redis.

Kafka, on the other hand, is a high throughput, distributed log that can be used as a queue. Here any number of users can produce and consumers can consume at any time they want. It also provides persistence for the messages sent through the queue.

Final Take:

Use Redis:

1. If you want a fire and forget kind of system, where all the messages that you produce are delivered instantly to consumers.
2. If speed is most concerned.
3. If you can live up with data loss.
4. If you don't want your system to hold the message that has been sent.
5. The amount of data that is gonna be dealt with is not huge.

Use kafka:

1. If you want reliability.
2. If you want your system to have a copy of messages that has been sent even after consumption.
3. If you can't live up with data loss.
4. If Speed is not a big concern.
5. data size is huge

Solution 2

Redis 5.0+ version provides the Stream data structure. It could be considered as a log data structure with delivery guarantees. It offers a set of blocking operations allowing consumers to wait for new data added to a stream by producers, and in addition to that, a concept called Consumer Groups.

Basically Stream structure provides the same capabilities as Kafka.

Here is the documentation https://redis.io/topics/streams-intro

There are two most popular Java clients that support this feature: Redisson and Jedis

Redisson provides ReliableTopic object if reliability of delivery is required. https://github.com/redisson/redisson/wiki/6.-distributed-objects/#613-reliable-topic

Comments

[David Anderson]

I'm not sure why this question was closed as "opinion-based"? There are objective technical differences between the two and the existing answer clearly outlines those differences.

[Zeni]

One main difference is that Redis Pub/Sub is push based while Kafka Pub/Sub is pull based. That means messages published to Redis will be automatically delivered to subscribers instantly, while in Kafka Data/messages are never pushed out to consumers, the consumer will ask for messages when the consumer is ready to handle the message. cloudkarafka.com/blog/… kafka.apache.org/documentation.html#design_pull

[David Dahan]

Reading this: redis.io/topics/persistence it seems possible to me to hold the messages that have been sent. Am I wrong?

[Younes]

@DavidD: The link you provided explians how you can configure redis to ensure that messages that have been sent but not processed yet will not be lost after a restart of redis. Although it is possible to do that, redis doesn't allow to hold (or keep to reuse words of @Karthikeyan) out of the box.

[mjs]

Nikita himself :) Elegant library! Just started using it. Well structured and thought out! You are a genius sir!

[mjs]

I am having some questions regarding proper use and not, and I am afraid of making the wrong assumptions? Perhaps you could review the two questions I've added here on SO. Also would love to add you on Skype to bother you sometimes if that is ok. I can provide some insight on how I desire to use it. Not a total noob :)

[mjs]

For instance, I am currently creating a cachable map ... using a runtimes id as key, and then adding a list of stuff that the system is currently processing from a deque ... the list, I can create an ArrayList for i guess, i believe redisson will convert it internally for me, but if I don't and create a redisslon list, then I have to give it a name, correct? What name would you give that list internally then? A random id? Should your API then not also provide a parameter less createList, createMap and etc since there is a usecase for it?

[mjs]

Sure I can send in a randomUuid but would be nice to know redisson has a good name generator. I am also writing my own Deque for processing batch jobs containing a redisson deque, backed up by a map containing "taken" elements. If we have 10 systems with each 8 threads processing the queue, and a nuclear bomb occurs, those would all be lost and left unprocessed, since they were taken but not completly processed.

[mjs]

My approach is to put the taken element inside a separate map keeping track of them using a system / runtime / pod id. Then I have to keep track of how long they have been there before i push them back to queue. It is about creating a bulletproof queue / deque. Does that make sense? I also have to do the take and put to map in a safe manner, and roll back take if put is not successful (ideally if the nuclear blast or electrical outage happens exactly in that moment, between take and put)

[mjs]

I think redisson could well provide such a feature built in that uses all these approaches to achieve a safe deque, worker mechanism.

[mjs]

Here are the 3 questions i've asked so far: stackoverflow.com/questions/63575219/… stackoverflow.com/questions/63567380/… stackoverflow.com/questions/63563961/… You do not have to answer them but you could perhaps share some insight or comments.

[Benargee]

OP did not ask for a popularity contest. They want to know which one is meant for which use case.