An Overview of Backup tools for Redis

Redis is a well-known in-memory datastore, quite commonly used for caching the data from a website or the database. You may think in such use cases backups of the data is not really relevant, but that’s not really the truth. In this short blog we are going to discuss the ways in which you can backup the data stored on your Redis cluster. Please keep in mind that even if we’ll be using “cluster”, we are going to write about Redis sentinel setup, not the “Redis Cluster” deployment.

Why do We Want to Have Backups of an In-memory Datastore Used for a Caching Layer?

First of all, let’s try to have a quick discussion about why backups are still relevant even if you plan to use Redis as a datastore for caching. Obviously, everything depends on the size of the cache and the size of the infrastructure you have, but let’s consider a decently-sized dataset. What is very important to keep in mind is that the cache layer is crucial for the operation of the database and application layers: it offloads the database from requests. If a result for a given request can be found in the cache, the result will be served from the cache instead of the database. In-memory cache, like what Redis provides, may serve several orders of magnitude more requests in the same time than the database itself. As you can imagine, sending the traffic to the cache layer instead of the database is quite important.

What will happen, though, if the cache layer doesn’t contain the result of a request? Let’s say the node that is being queried just recently crashed and has been started fresh. In that case, the details will depend on the implementation. In a proper one you will see requests piling up on the Redis node, while exactly one thread executes a query on the database layer in an attempt to populate the cache. In a not-so-proper implementation you may see those requests piling up against the database, slowing it to a crawl, as every connection that made an unsuccessful attempt to get the data from rollback to the direct connection to the database. One way or the other, the latency significantly increases as the requests have to wait for the database to provide the data.

There are many ways in which you can reduce the impact. For example, you could send just a small percentage of the traffic to a new Redis node, allowing it to warm up slowly while minimizing the impact to just a very small subset of the connections. Only when the node is fully warm you would let the regular share of traffic hit it. The warm-up period may take a while though, it is also not without an impact on the rest of the Redis cluster. As fast as an in-memory store is, taking one node out of a three-node cluster will result in a 50% workload increase for the remaining two nodes. This, at the minimum, will somewhat impact the latency for the duration of the time needed for the third node to warm up.

Here’s where the backup comes in. As long as the failure wasn’t taking too long and the data in the cache is not outdated (as typical cache implementation may have some sort of Time To Live limits for cache entries), you can warm up the node pretty much instantly by restoring the backup. Instead of waiting for all (or at least majority) of query types to populate the cache you can just restore the backup and have the same data in memory as before the crash. As you can imagine, this is a way to significantly increase the speed in which your Redis node may ramp up to speed and start serving its full share of the traffic.

Backup methods for Redis

As we established that having a backup of data stored in Redis has its advantages, now we are going to discuss some of the methods you can use to take the backup off a Redis sentinel cluster.

A Filesystem Snapshot

Redis, while it is an in-memory datastore, does come with a couple of methods to persist the data on disk. We are talking about append-only files or the RDB snapshots. This is intended to allow a Redis instance to recover from a crash and restore the data with either full durability (using Append Only File) or restore to the latest RDB snapshot, losing data that has been created in the meantime. Such behavior allows us to use snapshots of the filesystem to backup the data. Please keep in mind that, not only in the case of the Redis, database restoration process from a filesystem snapshot can be treated just like the crash of the node. Recovery process required to clean up old, partially executed transactions and restore the data is very similar in both restoration from the crash and starting from a filesystem snapshot.

The creation of a filesystem snapshot depends mostly on the environment that you use. A snapshot can be created on a volume level, if the cloud provider or the hardware or software used allows for that. VM’s can be snapshotted from the hypervisor level, some Network Attached Storage solutions also allow for the snapshots to be made on the disk array level. It is also possible to use a filesystem like ZFS – it will allow for the snapshots to be created as well.

RDB snapshot

Another option to take a backup of Redis would be to use the native backup method and create an RDB file. RDB files are the snapshots of the database, they can be triggered by hand or you can configure Redis to run a snapshot on a regular basis, depending on the time passed from the previous snapshot and on the number of updates to the Redis database. Keep in mind that having a copy of the RDB file is enough to restore the database. You can copy it to an external location (different node, different storage, different cloud provider) and use it to restore the data on a node should the problem arise. This is snapshot-based so all data from the time of the last snapshot till the time of a failure will be lost but it is still better to recover the majority of the data from the cache than to warm it up from the beginning.

Logical backup

The third way in which data stored in Redis can be backed up is through logical backup. Logical backup means simply that the data is stored in a text format compared to binary formats for filesystem snapshot or RDB snapshot. All of the datastores that can be queried (which, pretty much covers all of them, the point of a datastore is to be queried at some point) can be backed up in a logical way. There can be dedicated software like mysqldump or pg_dumpall or it can be a script written by the user to extract the data from the datastore and keep it in some format. The format depends on the tool, it can be JSON, it can be a list of native commands to insert the data into the database. Redis is not different. It does not come with a built-in solution but there are multiple scripts available in the GitHub that are intended to dump the data and then load it if needed. An example of such a script might be redis-dump or node-redis-dump.

The advantage of logical backups is that they are, at least to some extent, human-readable (even if some tools might be needed to help parse JSON) and can be modified to exclude some of the data or edit some of the entries you may have to edit.

As you can see, there are a couple of ways in which Redis datastore can be backed up. No matter which method you would like to pick, what is very important is to make sure that you actually take the backup and that the backup you took can be restored. Regular backup verification attempts are crucial to ensure the backups you stored are indeed available and working properly and can be used to recover your Redis instances.

If you have experience with backing up Redis, we would like to hear from you. Please make sure that you leave a comment below.