Redis master/slave replication - single point of failure?

Question

How does one upgrade to a newer version of Redis with zero downtime? Redis slaves are read-only, so it seems like you'd have to take down the master and your site would be read-only for 45 seconds or more while you waited for it to reload the DB.

Is there a way around this?

Solution 1

When taking the node offline, promote the slave to master using the SLAVEOF command, then when you bring it back online you set it up as a slave and it will copy all data from the online node.

You may also need to make sure your client can handle changed/missing master nodes appropriately.

If you want to get really fancy, you can set up your client to promote a slave if it detects an error writing to the master.

Solution 2

Redis Team has very good documentation on this

Core Steps:

• Setup your new Redis instance as a slave for your current Redis instance. In order to do so you need a different server, or a server that has enough RAM to keep two instances of Redis running at the same time.
• If you use a single server, make sure that the slave is started in a different port than the master instance, otherwise the slave will not be able to start at all.
• Wait for the replication initial synchronization to complete (check the slave log file).
• Make sure using INFO that there are the same number of keys in the master and in the slave. Check with redis-cli that the slave is working as you wish and is replying to your commands.
• Configure all your clients in order to use the new instance (that is, the slave).
• Once you are sure that the master is no longer receiving any query (you can check this with the MONITOR command), elect the slave to master using the SLAVEOF NO ONE command, and shut down your master.

Full Documentation:

Upgrading or restarting a Redis instance without downtime

Solution 3

You can use Redis Sentinel for doing this, the sentinel will automatically promote a slave as new master. you can find more info here http://redis.io/topics/sentinel.

Sentinel is a system used to manage redis servers , it monitors the redis master and slaves continuously, and whenever a master goes down it will automatically promote a slave in to master. and when the old master is UP it will be made as slave of the new master.

Here there will be no downtime or manual configuration of config file is needed. You can visit above link to find out how to configure sentinel for your redis servers.

Solution 4

Note, you may have to check and set the following config to write to your slave. ("Since Redis 2.6 by default slaves are read-only")

redis-cli config set slave-read-only no

-- Example

-bash-4.1$ redis-cli info
 Server
redis_version:2.6.9

-bash-4.1$ redis-cli slaveof admin2.mypersonalsite.com 6379
OK

-bash-4.1$ redis-cli set temp 42
(error) READONLY You can't write against a read only slave.

-bash-4.1$ redis-cli slaveof no one
OK

-bash-4.1$ redis-cli set temp 42
OK

-bash-4.1$ redis-cli get temp
"42"


-bash-4.1$ redis-cli config set slave-read-only no
OK

-bash-4.1$ redis-cli slaveof admin2.mypersonalsite.com 6379
OK

-bash-4.1$ redis-cli set temp 42
OK

-bash-4.1$ redis-cli get temp
"42"

View more solutions

Comments

[nornagon]

What happens to writes that were pending on the old master? What's going on in between the time when the old master drops offline and when all the slaves know about the new master?

[Tom Clarkson]

Those slaves will have out of date data until they are able to load data from the new master. However, you can set this up ahead of time - slaves don't have to connect directly to the master. For writes as the change is happening, set up your client to write to the new master before the old one goes down - slaves aren't actually read only, it's just that the writes won't get sent to other nodes, which doesn't matter in this scenario.

[talonmies]

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes.

[Abhi]

Well the solution for above problem is making use of "sentinel" , which is what is explained briefly in the above link, you need to understand sentinel in order to configure it for your redis. the link above provides the clear explanation for it.

[Florian Heigl]

This idea of 'really fancy' somehow predates VAXcluster.

[Enrico Marchesin]

Nice idea to recap doc here: just add the missing step "Allow writes to the slave using CONFIG SET slave-read-only no" before client reconfiguration. In recent Redis versions, slaves are read only by default.