How to design redis pub/sub for an instant messaging system?

Question

I am new to redis pub/sub. I have a chat facility in the system which is like IM. So I would like to use redis pub/sub. As I have examined the samples most of them are designed based on a chat room. In my system I will have multiple chat rooms between users like;

A:B
A:C
D:C
E:F

So, the lines above are the rooms. And I have implemented the server with node.js like below;

var store = redis.createClient();
var pub = redis.createClient();
io.sockets.on('connection', function (socket) {
    var sub = redis.createClient();

    sub.on("message", function(pattern, data){
            data = JSON.parse(data);
        socket.send(JSON.stringify({ type: "chat", key: pattern, nick: data.nickname, message: data.text }))
        }
    });

    socket.on('message', function (messageData) {
        store.incr("messageNextId", function(e, messageId) {
        var room = ""
        var from = messageData.clientId > socket.nickname ? socket.nickname : messageData.clientId;
        var to = messageData.clientId < socket.nickname ? socket.nickname : messageData.clientId;   
            room = from + ":" + to;

        var message = { id: messageId, nickname: socket.nickname, text: messageData.text };
        store.rpush("rooms:" + room, JSON.stringify(message), function(e, r) {  
             pub.publish(room, JSON.stringify(message))
        });
    });
});

As you can see I am creating a new redis subscriber for each connection. In other chat room samples redis subscriber client is created globally. And there exists only three connections all the time and that solves their problem because when a publisher publishes a message all connected clients should get it. But I have a constraint here. I want to open a chat session between two users and only these users should be the subscribers. The code above works as I would like to but I do not know if it is OK for redis to create a new subscriber client for each connection.

It would be great to hear your suggestions. Thanks in advance.

Solution

As always, you need to benchmark things like this for your own use-case -- it's not possible to give general advice. You might need to increase the maximum number of open files on your system, either system-wide or for the redis user. This also applies to the user running your web server, of course.

That said, you should make sure to listen for socket.on('disconnect') and quit() the redis subscriber when a user leaves. You might also be interested to know that socket.io has a redis backend, which leverages redis pub/sub, and it also has the concept of rooms, so you might save yourself some trouble by using that since you're already depending on socket.io.

Edit: After a quick check, I get this error message from Redis after 991 subscribers:

Ready check failed: Error: Error: ERR max number of clients reached

Here is from the default redis.conf:

# Set the max number of connected clients at the same time. By default
# this limit is set to 10000 clients, however if the Redis server is not
# able ot configure the process file limit to allow for the specified limit
# the max number of allowed clients is set to the current file limit
# minus 32 (as Redis reserves a few file descriptors for internal uses).
#
# Once the limit is reached Redis will close all the new connections sending
# an error 'max number of clients reached'.
#
# maxclients 10000

My system (Ubuntu 11.11) comes with a default nofile limit of 1024, so my quick test should fail after 992 connected clients, which seems about right from the test (I also have one client for the publisher). My suggestion to you is to inspect your nofile limit (on my system it's in /etc/security/limits.{conf,d/*} and your redis maxclients setting, and then benchmark, benchmark, benchmark!

Comments

[György Balássy]

Thanks for the detailed answer. I knew that socket.io has the room concept but I did not aware of it is using pub/sub behind the scenes. I will give a try on that. And sure, I will start the benchmark tests, currently I'm just trying to find a good start point. Thanks for the suggestions.

[Linus Thiel]

@AliErsöz: Great. If you find out my answer was satisfactory, please consider accepting it as the correct answer by clicking the big checkmark to the upper left.