blog

Installing Redis Master-Slave with Sentinel for Auto Failover

Agus Syafaat

Published July 5, 2021

In a previous blog, we already wrote on how to set up Redis Master-Slave replication with manual failover.

The disadvantages of Redis Master-Slave architecture, as mentioned in a previous blog, refer to the ability to do the automatic failover. We cannot rely on manual tasks to do failover: imagine if the Redis master shuts down during midnight and there is no failover – this will impact the application service as well.

Redis provides a high availability solution for redis service called Redis Sentinel. It is a monitoring and service discovery solution and also handles the failover for redis. In this blog, we will review how to set up Redis Master-Slave architecture for Auto Failover with Redis Sentinel.

What is Redis Sentinel?

Redis Sentinel is a high availability solution provided by Redis. There are various features in Redis Sentinel which we can utilize, such as monitoring, notification, automatic failover, and setting up configuration. Sentinel uses a quorum base for promoting the new master if the old master crashes. The quorum is the number that is configured on each Sentinel – it’s used to ensure that the master cannot be reachable and decide if the master has failed. The election leader will be done by Sentinel based on majority votes of the members.

For example, if we have 3 Redis Sentinel instances in different machines and we configure the quorum to two, then the failover will happen if two Sentinel processes agree that the Master node is unreachable, one of Sentinels will do the failover, elect the leader, and promote it to become the Master. The failover will not happen if the majority of Sentinels fail to communicate.

There are some parameters that need to be configured in /etc/redis/sentinel.conf such as:

```
sentinel monitor    
```
The above parameter needs to be commented out or configured in sentinel configuration. The explanation of the command is that it monitors the master node with specific ip addresses and redis ports (the default port is 6379) with the quorum number.
```
down-after-milliseconds
```
Defines the threshold amount (in milliseconds) for Sentinels categorizing the nodes into not reachable and not responsive to ping commands or give specific error responses during the checking process.
```
parallel-syncs
```
It is the value which defines the number of slaves configured to use the master after the failover happens.
```
failover-timeout
```
The parameter is used as a factor to give a time for Sentinel to failover the same master again when Sentinel voted another Sentinel for failover.

Sentinel Installation

The installation of Redis Sentinel is really straightforward, usually it will be included during the installation of Redis, but if the Sentinel is still not installed, we can install directly through the package manager. Currently we have 3 nodes for setting up the High Availability using Sentinel, the ip addresses are:

10.10.10.10 (Redis Master + Sentinel)
10.10.10.11 (Redis Slave + Sentinel)
10.10.10.12 (Redis Sentinel).

In this case, we use the Ubuntu operating system, just run the apt install command to install the Redis and Sentinel:

$ apt install redis-server
$ apt install redis-sentinel

Then we can check if the Redis and Sentinel is up and running:

redis-server.service - Advanced key-value store

   Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: ena

   Active: active (running) since Fri 2021-07-02 10:26:47 UTC; 4h 42min ago

     Docs: http://redis.io/documentation,

           man:redis-server(1)

  Process: 14288 ExecStop=/bin/kill -s TERM $MAINPID (code=exited, status=0/SUCCESS)

  Process: 14292 ExecStart=/usr/bin/redis-server /etc/redis/redis.conf (code=exited, sta

 Main PID: 14316 (redis-server)

    Tasks: 4 (limit: 1106)

   CGroup: /system.slice/redis-server.service

           └─14316 /usr/bin/redis-server 127.0.0.1:6379



Jul 02 10:26:47 n2 systemd[1]: Starting Advanced key-value store...

Jul 02 10:26:47 n2 systemd[1]: redis-server.service: Can't open PID file /var/run/redis/

Jul 02 10:26:47 n2 systemd[1]: Started Advanced key-value store.



redis-sentinel.service - Advanced key-value store

   Loaded: loaded (/lib/systemd/system/redis-sentinel.service; enabled; vendor preset: e

   Active: active (running) since Fri 2021-07-02 10:02:30 UTC; 5h 6min ago

     Docs: http://redis.io/documentation,

           man:redis-sentinel(1)

  Process: 14195 ExecStop=/bin/kill -s TERM $MAINPID (code=exited, status=0/SUCCESS)

  Process: 14199 ExecStart=/usr/bin/redis-sentinel /etc/redis/sentinel.conf (code=exited

 Main PID: 14224 (redis-sentinel)

    Tasks: 4 (limit: 1106)

   CGroup: /system.slice/redis-sentinel.service

           └─14224 /usr/bin/redis-sentinel *:26379 [sentinel]



Jul 02 10:02:30 n2 systemd[1]: Starting Advanced key-value store...

Jul 02 10:02:30 n2 systemd[1]: redis-sentinel.service: Can't open PID file /var/run/sent

Jul 02 10:02:30 n2 systemd[1]: Started Advanced key-value store.

There will be two background processes in the operating system, one is for Redis service and another one is for the Sentinel service. We can repeat the installation on each node.

Setting up Redis Sentinel

To configure and integrate the Sentinel with Redis Server, we need to add the parameters in sentinel.conf. Add below configurations in the sentinel configuration file on each Sentinel:

sentinel monitor redis-master 10.10.10.10 6379 2

sentinel down-after-milliseconds redis-master 1500

sentinel failover-timeout redis-master 3000

protected-mode no

Our Redis master is on ip address 10.10.10.10 and After the configuration was added, we needed to restart the Redis service and Sentinel service.

root@n2:~# systemctl restart redis-server
root@n2:~# systemctl restart redis-sentinel

Finally, after the configuration has been finished, we can verify that the Sentinels work by grabbing the heartbeat in the Sentinel logs:

14454:X 02 Jul 15:16:49.895 * +sentinel-address-switch master redis-master 10.10.10.10 6379 ip 10.10.10.12 port 26379 for cc9f9d652f2f870d1725d74fdd99d3edaf97294b

14454:X 02 Jul 15:16:49.895 * +sentinel-address-update sentinel cc9f9d652f2f870d1725d74fdd99d3edaf97294b 10.10.10.12 26379 @ redis-master 10.10.10.10 6379 1 additional matching instances

14454:X 02 Jul 15:16:51.742 * +sentinel-address-switch master mymaster 127.0.0.1 6379 ip 127.0.0.1 port 26379 for cc9f9d652f2f870d1725d74fdd99d3edaf97294b

14454:X 02 Jul 15:16:51.742 * +sentinel-address-update sentinel cc9f9d652f2f870d1725d74fdd99d3edaf97294b 127.0.0.1 26379 @ mymaster 127.0.0.1 6379 1 additional matching instances

14454:X 02 Jul 15:16:52.014 * +sentinel-address-switch master redis-master 10.10.10.10 6379 ip 10.10.10.12 port 26379 for cc9f9d652f2f870d1725d74fdd99d3edaf97294b

14454:X 02 Jul 15:16:52.015 * +sentinel-address-update sentinel cc9f9d652f2f870d1725d74fdd99d3edaf97294b 10.10.10.12 26379 @ redis-master 10.10.10.10 6379 1 additional matching instances

And check the replication information:

root@n1:/var/log/redis# redis-cli

127.0.0.1:6379> info replication

# Replication

role:master

connected_slaves:2

slave0_ip=10.10.10.12,port=6379,state=online,offset=2781219,lag=1

slave1_ip=10.10.10.11,port=6379,state=online,offset=2781503,lag=1

master_replid:075e651ef398ad202003105053091b4e6b43d323

master_replid2:0000000000000000000000000000000000000000

master_repl_offset:2781503

second_repl_offset:-1

repl_backlog_active:1

repl_backlog_size:1048576

repl_backlog_first_byte_offset:1732928

repl_backlog_histlen:1048576

127.0.0.1:6379>

What is Redis Sentinel?

Sentinel Installation

Setting up Redis Sentinel

Subscribe below to be notified of fresh posts