Redis Multi-Master Replication

Question

Scenario: We have two datacentres which run concurrently (both serve traffic).
Each has it's own entire stack of infrastructure, so it can operate without the other being up.

That said, we would like, when network conditions allow for it, for our Redis database to be synchronised between the two. Losing keys during a network partition is acceptable, as is having duplicate keys - it's only cache data. But, we get the most benefit from the cache when both datacentres are up (from 15-20% cache hits, to 30-40% cache hits).

After some searching around I couldn't find anything that would effectively give us multi-master. (Mostly saying "don't do that" or "it's not supported").

In the end, I wrote a client that connects to both masters, subscribes to keyspace events for the database, and then pushes all the SET commands between DBs (all our keys are set with expiry) - with some internal "recently seen" cache to prevent a replay loop.

At the moment, that's working great - the only real downside is that after we get a new key event, we then have to issue a GETEX to get the key and it's expiry - so you end up with a second read (thus about 2x the RTT latency for the remote DC). It's also limited to really just supporting SET.

My question is: Is there a better way of getting this kind of multi-master replication?

I started looking at SYNC/PSYNC, but there's not a lot of documentation on these protocols, and not sure what obligations a client might have to not break the server.

Solution

I am also looking for a solution to this challenge, so far, I found: https://github.com/Netflix/dynomite/wiki/Architecture And maybe... https://github.com/CodisLabs/codis or https://github.com/twitter/twemproxy

Comments

[Admin]

Unfortunately none of these support the full level of commands, and for the most part don't do the 'best effort' idea, they're more clustering options than anything else.