blog
Backing Up Redis Clusters: A Guide
As we discussed in one of our previous blogs, backing up Redis is quite important to do, even if all we use Redis for is a cache for your application or the database. We have discussed some different ways in which you can backup Redis sentinel clusters. We talked about filesystem snapshots, copying RDB file or taking logical backups. In this blog post let’s take a look at what are the steps to create a backup of Redis cluster and how you would restore such backup if needed.
Backing up the Redis sentinel cluster
First of all, let’s talk about backing up the data. We are going to use the simplest method available – we will just take a copy of the Redis directory contents. Please keep in mind that all the steps we will show are executed on a test setup without any kind of vulnerable data. Reproducing this on production should require proper password and privilege management, backup encryption at rest and in transit and so on.
Create a new RDB snapshot
RDB snapshots can be created on the fly using SAVE or BGSAVE commands. The former is quite impactful as it halts all other operations on the cluster for the duration of the data dump. The latter is more elegant – Redis spawns a child process that is dedicated to create the dump while the parent process still responds to all requests that may come in the meantime. To make it even less impactful, it is possible to set a CPU affinity for both parent and child processes and ensure that they run on separate CPU cores, not competing for resources. Below you can find options revolving around CPU affinity that can be configured in Redis:
127.0.0.1:6379> CONFIG GET *cpulist*
1) "server_cpulist"
2) ""
3) "bio_cpulist"
4) ""
5) "aof_rewrite_cpulist"
6) ""
7) "bgsave_cpulist"
8) ""If we want to run a snapshot, the best way would be to run the BGSAVE command.
127.0.0.1:6379> BGSAVE
Background saving startedCopy RDB file to a backup directory
When the snapshot starts, in Redis log we should see lines like:
250396:M 25 Jun 2021 09:06:51.548 * Background saving started by pid 552705
552705:C 25 Jun 2021 09:06:52.430 * DB saved on disk
552705:C 25 Jun 2021 09:06:52.431 * RDB: 1 MB of memory used by copy-on-write
250396:M 25 Jun 2021 09:06:52.464 * Background saving terminated with successThis is an indication that the snapshot was created and we can copy the files to a separate location.
root@vagrant:~# cp /var/lib/redis/dump.rdb /backup/
root@vagrant:~# ls -alh /backup/
total 4.4M
drwxr-xr-x 2 root root 4.0K Jun 25 09:08 .
drwxr-xr-x 21 root root 4.0K Jun 25 09:08 ..
-rw-r----- 1 root root 4.4M Jun 25 09:08 dump.rdbThis is pretty much it. If we wanted, we could also copy the Append Only File, but it is not required. We can think of AOF file as WAL segments or binary logs that can be applied on a database to restore to the latest state. If we have an RDB file, this is the latest, consistent state at the time of the backup so we are all good.
RDB file is already compressed, at least by default. You can check the configuration on your Redis server:
127.0.0.1:6379> CONFIG GET rdbcompression
1) "rdbcompression"
2) "yes"If you want, you may now encrypt the file.
Restoring backup
Restore with AOF disabled
Taking backups is just the beginning. The most important part will be the restore process. Ideally, each backup goes through the process of verification, restoring it on a separate node, making sure that the data can be accessed. Let’s go step by step through such process.
First, we have to have a free Redis instance to test the restore on. Once we have it, we have to copy the backup file to its destination:
root@vagrant:~# scp /backup/dump.rdb 10.0.0.184:/restore
dump.rdb
100% 4461KB 5.6MB/s 00:00Then, on the target host, we have to stop the Redis server if it is running, copy the file to the Redis data directory, change the owner of the file to Redis user and then start the Redis instance.
root@vagrant:~# service redis stop
root@vagrant:~# cp /restore/dump.rdb /var/lib/redis/
root@vagrant:~# chown redis.redis /var/lib/redis/
appendonly.aof dump.rdb
root@vagrant:~# chown redis.redis /var/lib/redis/dump.rdb
root@vagrant:~# service redis startThis will be enough to get the data loaded at the startup. In the Redis log file you should see entries like:
234868:M 25 Jun 2021 10:14:16.217 * Loading RDB produced by version 6.0.6
234868:M 25 Jun 2021 10:14:16.217 * RDB age 3983 seconds
234868:M 25 Jun 2021 10:14:16.217 * RDB memory usage when created 21.69 Mb
234868:M 25 Jun 2021 10:14:16.386 * DB loaded from disk: 0.170 seconds
234868:M 25 Jun 2021 10:14:16.387 * Ready to accept connectionsRestore with AOF enabled
If your Redis node has Append Only File enabled, the steps we described above require a bit of change. If AOF is enabled, Redis will ignore RDB files and it will rely on AOF to recover. Therefore, before we attempt to restore RDB backup, we have to set ‘appendonly’ to ‘no’ in the Redis configuration file. Then proceed as above, stopping it, copying RDB file and starting again.
Once the database started and backup is restored, you should re-enable AOF by reverting the change in the configuration file. Then you should create a new AOF file and enable AOF in runtime:
127.0.0.1:6379> BGREWRITEAOF
Background append only file rewriting started
127.0.0.1:6379> CONFIG SET appendonly yes
OKThis is all that you need to have a RDB backup restored on the new Redis instance with AOF enabled.
How to backup Redis cluster using ClusterControl
We are always working to implement support for more datastores in ClusterControl. Currently our developers are focussing on adding Redis to the list of technologies supported by our product. We believe that Redis fits great in the lineup of Open Source databases supported by ClusterControl. Let’s take a look at how the ClusterControl handles backups of Redis sentinel cluster.
As we mentioned, this is a development build so details may and most likely will change. More features will be added, UI will be also created. As of now only CLI is available.
root@vagrant:~# s9s backup --create --cluster-id=2 --log
Using SSH credentials from cluster.
Cluster ID is 2.
The username is 'root'.
Elected host 10.0.0.181:6379 for creating the backup.
10.0.0.181: Creating backup dir '/root/backups/BACKUP-2'.
10.0.0.181: Backup command replied: OK.
No custom script is specified in etc/cmon.d/cmon_2.cnf
10.0.0.181: Backing up the rdb file: /var/lib/redis/dump.rdb
10.0.0.181: Backing up the aof file: /var/lib/redis/appendonly.aof
No custom script is specified in etc/cmon.d/cmon_2.cnf
10.0.0.181:/root/backups/BACKUP-2: Preferring the global retention setting: 31 days (be deleted on 2021-07-26).
Created symlink '/root/backups/LATEST' -> '/root/backups/BACKUP-2'
Checking for backup retention (clearing old backups).
Local backup retention is 31 day(s).
Cloud backup retention is 180 day(s).
Kept safety backup copies 1.
Querying records older than 31 day(s).
Checking for backups to purge.
No old backup records found, nothing to delete.As you can see, ClusterControl backs up both RDB and AOF files so you can restore the data using either of them. Restoring a backup is also supported. First, we have to check what backups are available and what are their ID’s:
root@vagrant:~# s9s backup --list --long
ID PI CID V I STATE OWNER HOSTNAME CREATED SIZE TITLE
1 - 2 - F COMPLETED dbu 10.0.0.181 2021-06-23 12:18:28 5769184 BACKUP-1
2 - 2 - F COMPLETED dbu 10.0.0.181 10:31:18 55762191 BACKUP-2
Total 2Then we just pick the backup we want to restore and that’s all:
root@vagrant:~# s9s backup --restore --cluster-id=2 --backup-id=1 --log
Using SSH credentials from cluster.
Cluster ID is 2.
The username is 'root'.
10.0.0.181: Stopping redis-sentinel.
10.0.0.181:26379: Redis Sentinel is stopped by the user.
10.0.0.182: Stopping redis-sentinel.
10.0.0.182:26379: Redis Sentinel is stopped by the user.
10.0.0.183: Stopping redis-sentinel.
10.0.0.183:26379: Redis Sentinel is stopped by the user.
10.0.0.181: Stopping redis-server.
10.0.0.181:6379: Redis Server is stopped by the user.
10.0.0.182: Stopping redis-server.
10.0.0.182:6379: Redis Server is stopped by the user.
10.0.0.183: Stopping redis-server.
10.0.0.183:6379: Redis Server is stopped by the user.
Started a check at 2021-06-25 10:34:38 to determine if expected state STOPPED is stable.
Held expected cluster state STOPPED for 5 seconds.
10.0.0.181: Creating backup dir '/root/backups/BACKUP-1/BACKUP-4'.
10.0.0.181: Backup command replied: .
No custom script is specified in etc/cmon.d/cmon_2.cnf
10.0.0.181: Backing up the rdb file: /var/lib/redis/dump.rdb
10.0.0.181: Backing up the aof file: /var/lib/redis/appendonly.aof
No custom script is specified in etc/cmon.d/cmon_2.cnf
10.0.0.181: Restoring from '10.0.0.181':'/root/backups/BACKUP-1/dump.rdb'
10.0.0.181: Restoring from '10.0.0.181':'/root/backups/BACKUP-1/appendonly.aof'
Backup is successfully restored,
10.0.0.181: Starting redis-sentinel.
10.0.0.182: Starting redis-sentinel.
10.0.0.183: Starting redis-sentinel.
10.0.0.181: Starting redis-server.
10.0.0.182: Starting redis-server.
10.0.0.183: Starting redis-server.
Started a check at 2021-06-25 10:34:55 to determine if expected state STARTED is stable.
Held expected cluster state STARTED for 30 seconds.That’s it, the backup is restored on the “master” node and the remaining nodes from the cluster will replicate the current state of the database.
As you can see, creating backups of Redis cluster is not very complex, although it has some catches that you should be aware of. In the future you will not have to be bothered with it as ClusterControl will be handling the Redis backups for you.