Using BitTorrent Sync to Transfer Database Backups Offsite

BitTorrent Sync is a simple replication application providing encrypted bidirectional file transfers that can run behind NAT and is specifically designed to handle large files. By leveraging the simplicity of Bittorrent Sync, we can transfer backup files away from our cluster, enhancing the backups availability and reducing the cost of broken backup, where you can regularly verify your backups off-site. The bidirectional replication allows us to manage on-site backups remotely as changes happen on the other end (delete, rename, move) will be replicated to the other end as well.

What is BitTorrent Sync?

BitTorrent Sync, or BTSync, is an encrypted peer-to-peer file transfer application which uses the BitTorrent protocol to keep files on several clients synchronized. It is designed to transfer large files at the maximum speed supported by the network. 

Setting up a client is easy with a single application to install, selecting a folder via the built-in web application with a randomly generated or user-chosen 20 bytes secret which is used to sync that folder across clients.

To summarize:

• simple installation and configuration
• easy to use built-in web application
• easy to setup cross-sync or one-way sync
• add a new client by simply adding the shared folder(s) with the specified secret
• use a master secret (clients can read & write), read only or a one-time only secret
• encryption built-in, AES cipher and with a 256-bit key
• files larger than 4MB are synced in blocks of 4MB when there are any changes
• file versioning (keep an old version(s) of the file)

btsync vs rsync

Rsync is the most popular remote copying software which widely used for backup purposes and mirroring. Here are some comparisons between btsync and rsync:

• btsync is using many-to-many file transfer protocol while rsync is strictly one-to-one protocol 
• btsync runs well behind firewall and NAT environment. Dynamic IP is not a problem
• btsync is a cross-platform application. It can run on Windows, Linux, Mac, IOS, Android, FreeBSD on ARM/PPC/i386/x86_64 architectures
• btsync scales better than rsync
• btsync is truly server-less and distributed, while rsync is server-based and centralized

Example Deployment

We have scheduled our daily backups in ClusterControl to run at 3 AM. The backups will be stored in ClusterControl node, located under /storage/backups. We are going to set up BitTorrent Sync to replicate this directory on our backup server in office through WAN connection.

Following figure illustrates the target architecture that we are going to set up in this post:

Setting Up btsync on ClusterControl

1. Download the application from BitTorrent Labs and extract it into /usr/bin directory:

$ wget --content-disposition http://download-lb.utorrent.com/endpoint/btsync/os/linux-x64/track/stable
$ tar -xzf btsync_x64.tar.gz -C /usr/bin/

2. Generate a secret:

$ btsync --generate-secret
A2XOKT5RC2XEPEVB7SZDBGTLDANSCLDF4

3. Create the shared folder and PID directory:

$ mkdir -p /storage/backups
$ mkdir -p /var/run/btsync

4. Create a new configuration for btsync at /etc/btsync.conf and add following lines:

{
    "device_name": "ClusterControl",
    "listening_port": 55555,
    "storage_path": "/root/.sync",
    "pid_file": "/var/run/btsync/btsync.pid",
    "check_for_updates": true,
    "use_upnp": true,
    "download_limit": 0,
    "upload_limit": 0,
    "shared_folders": [
        {
            "secret": "A2XOKT5RC2XEPEVB7SZDBGTLDANSCLDF4",
            "dir": "/storage/backups",
            "use_relay_server": true,
            "use_tracker": true,
            "use_dht": false,
            "search_lan": true,
            "use_sync_trash": true
        }
    ]
}

5. Start the btsync process:

$ btsync --config /etc/btsync.conf

6. ClusterControl has ability to monitor custom process under Manage >> Process. Let’s configure it so ClusterControl will monitor the btsync process:

Setting Up btsync on Backup Server

1. Download the application and extract it into /usr/bin directory:

$ wget --content-disposition http://download-lb.utorrent.com/endpoint/btsync/os/linux-x64/track/stable
$ tar -xzf btsync_x64.tar.gz -C /usr/bin/

2. Create the shared folder and PID directory:

$ mkdir -p /storage/backups
$ mkdir -p /var/run/btsync

3. Create a new configuration for btsync at /etc/btsync.conf and add following lines:

{
    "device_name": "Backup Server",
    "listening_port": 55555,
    "storage_path": "/root/.sync",
    "pid_file": "/var/run/btsync/btsync.pid",
    "check_for_updates": true,
    "use_upnp": true,
    "download_limit": 0,
    "upload_limit": 0,
    "shared_folders": [
        {
            "secret": "A2XOKT5RC2XEPEVB7SZDBGTLDANSCLDF4",
            "dir": "/storage/backups",
            "use_relay_server": true,
            "use_tracker": true,
            "use_dht": false,
            "search_lan": true,
            "use_sync_trash": true
        }
    ]
}

Take note that we are using the same secret value as generated in ClusterControl server.

4. Start the btsync process:

$ btsync --config /etc/btsync.conf

As for the backup server, you could create a custom init.d script or defined the start command under /etc/rc.local to allow the process to start on boot. For more information you can refer to BitTorrent Sync user guide which you can download it here.

If you don’t want to mess with configuration files you can easily use the web application to set up the folders to share by adding the following to the configuration file:

  "webui" :
  {
    "listen" : "0.0.0.0:8888",
    "login" : "admin",
    "password" : "password"
  }

Web UI will still be disabled if you set shared_folders in config file. So, you need to remove that corresponding shared_folders array and make sure the config file is a valid JSON.

That’s it. Simple yet secure. Your backup files will be replicated over to the backup site within minutes.