Redis Sentinel or High Availability of Redis

Redis Sentinel:

Redis Sentinel is designed to monitor Redis Instance. It can automatically switch your master if it goes down or having some issue. Hence it will reduce the time for manual issue identification and switch. That will give us high availability,  that is why some people call it HA for Redis.

As per Redis team Sentinel does 4 things: Monitoring, Notification, Configuration Provider and Automatic failover. Redis Sentinel is used to monitor Redis Master Slave setup. If Redis Sentinel find a master is not working as expected then it starts a failover process, where it promotes the slave to master. In nutshell you can say Redis Sentinel will switch the master. All the other slaves are reconfigured or updated to use new master, by communicating new config. Application using Redis server is also updated about new configurations. We can also use Sentinel as service discovery, client can directly connect with Sentinel and ask for current master address. It will return new address in case of fail-over.

Redis Sentinel Distributed Nature:

Redis Sentinels are distributed in nature and following conditions are true for Redis Sentinel.

•    A cluster of Sentinel can failover a master even if some Sentinel are failing.
•    A single Sentinel not working well, or not well connected, can’t failover a master without authorization from other Sentinel.
•    Clients can connect to any random Sentinel to fetch the configuration of a master.

Keyword

You may get confused between Redis, Redis Server and Redis Sentinel. Redis Sentinel is packed in Redis 2.8, so Redis Sentinel is not a separate software or package. For clarity i am defining the terms used…

Redis2.8 refers to Redis package only. It contains Redis Server as well as Redis Sentinel.

Redis Server or Redis Master Slave means machine having some Redis package running Redis Servers.

Redis Sentinel Or Sentinel refers to Sentinel and is part of Redis package(majorly Redis 2.8).

Current Version of Redis Sentinel

“Sentinel 2”, is latest version as of today. Redis Sentinel 2 is shipped in Redis 2.8(latest version of Redis as of today).  “Sentinel 1” is buggy and advised not to used. That doesn’t mean you can’t use new “Sentinel 2” with Redis 2.6 Server or below.  You can install Redis 2.8 on Sentinel machines and let them monitor your Redis 2.6 server. But it will be good if you can migrate all your Redis Servers to Redis 2.8. Summary: Use Redis 2.8 to run Sentinel.

How to Run Redis Sentinel?

Redis Sentinel are very easy to setup. Here i assume that that you already have Redis Master Slave setup. if not, you can read here.

Redis Sentinel Config

Redis package have a file called sentinel.conf that contains documentation with examples. You can delete all the content of sentinel.conf and add below lines to it.

First 6 lines of sentinel conf file are self explanatory and defines the configs of Sentinel daemon. We will explain other lines one by one below:

  • First line “sentinel monitor” tells Sentinel to monitor a master and name it “MYMASTER”(name doesn’t matter). It have the address “127.0.0.1” and port “6379”, with quorum 2. Quorom 2 means Sentinel can start a failover only if at-least 2 Sentinel fails. In nutshell, “start failover, if more than two Redis Sentinel has detected that master is failed”.
  • Second line defines the password of Redis instance “MYMASTER”, here we are assuming it as “abc123” .
  • Third “down-after-milliseconds” is the time after which if an instance is not reachable (does not reply to the PINGs or reply with an error)  from a Sentinel. Then it can start thinking it is down. Sentinel will mark this instance as down and communicate to others. If other Sentinel says same and count of Sentinel is equal or more than Quorom defined in first line and it can start the failover process.
  • Fourth “parallel-syncs”,  number of slaves that be reconfigured as master.
  • Last “failover-timeout” is the wait time to fail same master again.

If you noticed we only defined the master address in config file. Here the Sentinel will fetch the slave data from master machine itself.

Start Redis Sentinel:

Any of the above method will work, but sentinel.conf is mandatory otherwise it will through an error. By default Sentinel run on 26379 port but you can change it in config file.

Now you need to do the same thing on 2 more machines, that will end up a cluster of 3 nodes. All the Sentinel will coordinate with each other automatically. You don’t need to define the communication between them. Make sure the ports are open for communication.

Important Setting:

If you are using password in Redis Master slave setup, don’t forget to define “masterauth” variable in “redis.conf” of both master and slave. When your down master will come up again, this is required. Otherwise your old master(down master) will not be able to connect with new master.

Check Redis Sentinels logs you get more idea, how it works:

Redis Sentinel Instance logs

In Above image Redis Sentinels version 2.8.17 in running on port 26379 with PID 22649. Sentinel is monitoring Redis master running on 127.0.0.1:6379. Last line says another instance of Sentinel is up and running on port 127.0.0.1:26380.

Redis Sentinel Logs for second instance

Logs of Second instance are exactly similar to as explained above.

For more details read Redis sentinel document:

http://redis.io/topics/sentinel