Using socat on CentOS/RHEL for Streaming Backups

There are several ways to perform remote-to-remote data copy – socat, netcat, rsync, scp, sftp. ClusterControl was using netcat for remote copy purposes, however we have flipped that decision to use socat. This blog post describes the reasons behind it.

A Brief History

ClusterControl introduced support for streaming backup back since v1.2.9 using netcat, replacing the older backup-and-copy method using secure copy (scp). Streaming backup is a much better option because it doesn’t consume disk space on the backup node and streaming happens on-the-fly, consuming only the disk space of the remote destination where the backup files are to be stored. Secure copy, on the other hand, requires setting up proper SSH keys and encryption adds performance overhead to the transfer process. This is unnecessary if you already have an isolated network for the database servers (encryption is also possible with socat or ncat).

Netcat is a better way of streaming files over the network due to its lightweight process, simpler implementation and speed if compared to rsync, scp or sftp. By default, ClusterControl uses port 9999 for this purpose, which is configurable via ClusterControl -> Backup page if the Storage Location dropdown is “Store on Controller”:

Introduction to netcat/socat

Netcat (also known as nc) is a computer networking utility for reading from and writing to network connections using TCP or UDP. It is quite simple to build a very basic client/server model using nc. On one console, start nc listening on a specific port for a connection. For example:

$ nc -l 1234

Netcat is now listening on port 1234 for a connection. On a second console (or a second machine), connect to the machine and port being listened on:

$ nc 127.0.0.1 1234

There should now be a connection between the ports. Anything typed in the second console will be concatenated to the first, and vice-versa. After the connection has been set up, nc does not really care which side is being used as a server and which side is being used as a client. The connection may be terminated using an EOF (‘^D’).

There are currently three most popular versions of netcat for RHEL based operating system:

• netcat (also known as nc) – The original project. Default in CentOS/RHEL 6.
• nmap-ncat (also known as ncat) – A better version of netcat developed and maintained by the nmap project. It has support for multiple protocols, simultaneous connections and SSL. Default in CentOS/RHEL 7.
• socat – Also a better version of netcat similar to nmap-ncat. Available in EPEL for CentOS/RHEL.

There are also a rewritten version of GNU’s and OpenBSD’s version which support additional features. For example, OpenBSD’s netcat supports TLS.

Netcat Issue on CentOS/RHEL

To create a backup on a database node and store it remotely using netcat, ClusterControl triggers the backup command on the target database node and streams the output by initiating two endpoints on source and destination hosts:

$ nc -dl 9999 # on destination (the storage node)
$ {backup command} | nc 192.168.100.10 9999 # on source (the backup node)

However, the program “nc” in CentOS/RHEL 7 is an alias of nmap-ncat, which is a different version of the original netcat:

$ ls -l /usr/bin/nc
lrwxrwxrwx 1 root root 4 Feb 13  2017 /usr/bin/nc -> ncat

On CentOS/Red Hat 6, “nc” is the original netcat project. Both are practically similar, however the command line options are not compatible between each other. Consider the following command to start the listening host for standard netcat:

$ nc -dl 9999

If the netcat version is the nmap-ncat version, which is the default installed on CentOS/RHEL 7, you would see the following error instead:

Ncat: Invalid -d delay "l" (must be greater than 0). QUITTING.

The -d in nmap-ncat is a delay flag option, which in netcat means detach from stdin. Depending on the options used, this could potentially break the communication between nodes if they are running on different CentOS/RHEL version. One solution is to detect the installed netcat version, whether it’s nmap-ncat or netcat and execute the correct listening command. However, ClusterControl prefers to use socat instead, as described in the next section.

Workaround

ClusterControl automatically favors socat if it is installed. Despite not being part of the standard OS package repository, you can have it installed via EPEL repository or manual installation from Socat official page. Socat is also a dependency package for Percona XtraDB Cluster which you can verify by using the following command:

$ yum deplist Percona-XtraDB-Cluster-server-57
...  
dependency: socat
...

Starting from ClusterControl v1.4.2, all database deployments performed by ClusterControl will have socat installed. ClusterControl will start socat to listen on the destination backup host with the following command:

$ socat -u tcp-listen:9999,reuseaddr stdout > {destination path on receiver host}

While on the source host (the host where the backup is performed), the following backup command is executed (depending on the backup options configured via ClusterControl UI):

$ ulimit -n 256000 && LC_ALL=C /usr/bin/innobackupex --defaults-file=/etc/my.cnf --galera-info --parallel 1 --stream=xbstream --no-timestamp . | gzip -6 - | socat - TCP4:192.168.1.100:9999

By using socat, ClusterControl doesn’t need to verify what type of netcat version is installed or need to fire different commands for each version. It also allows us to use more advanced options like reuseaddr. This improves the usability of the streaming process across multiple operating system versions.

We hope that will be of help to anyone out there trying to automate their backup streaming process.

Happy clustering!

PS.: To get started with ClusterControl, click here!