Redis WATCH MULTI EXEC by one client

Question

I am using NodeJS + Express + Redis on RedisOnGo + node_redis as a client. I expect a lot of concurrency, so trying to test WATCH. This example won't contain Express, just necessary stuff.

var redis = require("redis")
var rc = redis.createClient(config.redis.port, config.redis.host)

rc.auth(config.redis.hash, function(err) {
    if (err) {
        throw err
    }
})

rc.on('ready', function () {
    rc.set("inc",0)
    for(var i=1;i<=10;i++){
        rc.watch("inc")
        rc.get("inc",function(err,data){
            var multi = rc.multi()
            data++ // I do know I can use rc.incr(), this is just for example
            multi.set("inc",data)
            multi.exec(function(err,replies){
                console.log(replies)
            })
        })
    }
})

Expecting result: getting N errors in exec callbacks and finally getting "inc" variable = 10-N.

Unexpected result: getting 0 errors in exec callbacks but finally getting "inc" variable = 1.

Watch doesn't work with my code.

I have found this thread redis and watch + multi allows concurrent users. They say it is because of the only redis client.

Then I found this thread Should I create a new Redis client for each connection?. They say that generating a new client for each transaction "is definitely not recommended". I am lost.

Please also note, that I have to authenticate to Redis server. Thanks in advance!

EDITION 1:

I was able to make it work using local Redis instance (so I do not use client.auth) by creating a new client connection before each WATCH-MULTI-EXEC iteration. Not sure if it is good though, but results now are 100% accurate.

EDITION 2 Made it work if I create a new client connection before each WATCH-MULTI-EXEC iteration and then do client.auth and wait for client.on.

The question still exists, is it OK that I create new client connections for each iteration?

Solution 1

Your result is entirely predictable. And rightly so.

Keep in mind - node.js is one thread application. Node.js use asynchronous input-output, but the commands should be sent in redis strictly sequential "request-response". So your code and your requests executed strictly parallel while your are using just one connection to redis server.

Look at your code:

rc.on('ready', function () {
    rc.set("inc",0)
    for(var i = 1; i <= 10; i++){
        rc.watch("inc")
        //10 times row by row call get function. It`s realy means that your written
        //in an asynchronous style code executed strict in series. You are using just
        //one connection - so all command would be executed one by one.
        rc.get("inc",function(err,data){
            //Your data variable data = 0 for each if request.
            var multi = rc.multi()
            data++ //This operation is not atomic for redis so your always has data = 1
            multi.set("inc",data) //and set it
            multi.exec(function(err,replies){
                console.log(replies) 
            })
        })
    }
})

To confirm this do this steps:

1. Connect to redis and execute monitor command.
2. Run your node.js application

The output would be

    SET inc 0
    WATCH inc

    GET inc 
    .... get command more 9 times

    MULTI
    SET inc 1
    EXEC
    .... command block more 9 times

So that you get exactly the results that you wrote above: "getting 0 errors in exec callbacks but finally getting "inc" variable = 1.".

Is it OK that you create new client connections for each iteration?

For this sample - yes, its solves your problem. In general - it depends on how many "concurrent" query you want to run. Redis is still one threaded so this "concurrent" means just way to concurrent command batch to redis engine.

For example, if use 2 connections the monitor could give something like this:

 1 SET inc 0 //from 1st connection
 2 WATCH inc //from 1st connection
 3 SET inc 0 //from 2nd connection            
 4 GET inc //from 1nd connection            
 5 WATCH int //from 2nd connection       
 6 GET inc //from 2nd connection                 
 7 MULTI //from 1st connection           
 8 SET inc 1 //from 1st connection    
 9 MULTI //from 2nd connection           
10 SET inc 1 //from 2nd connection           
11 EXEC //from 1st failed becouse of 2nd connection SET inc 0 (line 3) 
        //was executed after WATCH (line 2) 
12 EXEC //success becouse of MULTI from 1st connection was failed and SET inc 1 from first 
        //connection was not executed

-------------------------------------------------------------------------------> time 
               |   |    |  |   |     |     |    |   |     |    |         |
connection 1  set watch | get  |     |   multi set  |     |   exec(fail) |
connection 2          set    watch  get            multi set            exec

Its very important to understand how redis execute your commands. Redis is single threaded, all command from all connection executed one-by-one in a row. Redis does not guarantee that command from one connection would be executed in a row (if here is another connections present) so your should MULTI if want be sure that your commands executed one block (if need it). But why WATCH needed? Look at my redis commands above. You can see that command coming from different connections are mixed. And watch allow you to manage this.

This beautifully explained in the documentation. Please read it!

Solution 2

I got your question finally.

If you want to test WATCH for concurrency, I think you need to change your code. as we know. WATCH only monitor changing of value, not getting value operation. so in your current code, all your get command will be executed successfully and get 0, then they will set inc to 1. all the set value are the same (1), so watch won't fail.

In the case, we need to ensure not only write operation is protected, but also read. before you set inc, you need to watch and modify another key which is as a pessimistic lock, and then we can get and change inc. in this way, it will make sure your expectation.

rc.set("inc",0)
for(var i=1;i<=10;i++){
    rc.watch("inc-lock")
    rc.get("inc",function(err,data){
        var multi = rc.multi()
        data++
        multi.incr("inc-lock")
        multi.set("inc",data)
        multi.exec(function(err,replies){
            console.log(replies)
        })
    })
}

I tested it in my PC.

[2013-11-26 18:51:09.389] [INFO] console - [ 1, 'OK' ]

[2013-11-26 18:51:09.390] [INFO] console - [ 2, 'OK' ]

[2013-11-26 18:51:09.390] [INFO] console - [ 3, 'OK' ]

[2013-11-26 18:51:09.390] [INFO] console - [ 4, 'OK' ]

[2013-11-26 18:51:09.391] [INFO] console - [ 5, 'OK' ]

[2013-11-26 18:51:09.391] [INFO] console - [ 6, 'OK' ]

[2013-11-26 18:51:09.392] [INFO] console - [ 7, 'OK' ]

[2013-11-26 18:51:09.392] [INFO] console - [ 8, 'OK' ]

[2013-11-26 18:51:09.393] [INFO] console - [ 9, 'OK' ]

[2013-11-26 18:51:09.393] [INFO] console - [ 10, 'OK' ]

Solution 3

If you want to use transaction/atomic MULTI operations but you want to do so using a shared connection, as far as I know your only option is using LUA.

I use LUA scripting within redis for a number of things, and the thing with LUA is the whole script will execute atomically, which is quite convenient. You have to be aware though that this means if you have a slow LUA script you are making redis slow for everyone using your server.

Also, when using LUA even if you can operate on different keys, be aware if you use more than one key in your script you won't be able to use Redis cluster once it is released. This is due to, when using a cluster, keys will be distributed to different Redis processes so your LUA script might not have access to all of them on a single server.

In any case, the problem with redis cluster would be the same when issuing a MULTI, since MULTI won't be allowed to set different keys on a cluster.

Cheers,

j

Comments

[igorpavlov]

Thanks for the answer. However, it does not cover my problem. I understand the queuing process of Redis, the problem here is quite straightforward though: I have a single node process and if I connect to Redis one time at the start of the application and try to use this connection to all queries - WATCH doesn't seem to work. If I create new connection each time I do the "transaction", WATCH works and actually watches the key. But I do not want to connect to Redis each time I need to make a request to it. That is the problem.

[igorpavlov]

Thanks again for your time, but still, not correct. In code I have written a comment "// I do know I can use rc.incr(), this is just for example". I do know incr() is an atomic operation. BUT, inside this code there can be a lot of different non-atomic operations like GET->SET. Also, you do not need to use WATCH with incr().

[igorpavlov]

Thank you, your answer seems to be quite explanatory and you have understood my question correct. If there are no more answers in 2 days, bounty is yours.

[Henry Leu]

I guess you don't get what my code mean. I incr inc-lock key for adding a watch lock. It doesn't matter our real inc logic. Two things. Btw, my code will issue error when in cluster env which can make real race condition. It will prove 'watch' is safe

[igorpavlov]

LUA is definitely a solution here, thanks. Not sure who will receive a bounty now =)

[Aebsubis]

I've executed your code with this extra command: setTimeout(function(){ rc.get("inc",function(err,data){ console.log('inc value: ' + data); }); },3000); And the result is [ 1, 'OK' ] [ 2, 'OK' ] [ 3, 'OK' ] [ 4, 'OK' ] [ 5, 'OK' ] [ 6, 'OK' ] [ 7, 'OK' ] [ 8, 'OK' ] [ 9, 'OK' ] [ 10, 'OK' ] inc value: 1 I was expecting inc value: 10