blog
Top Backup Tools for PostgreSQL
PostgreSQL has the reputation of being rock solid from its beginnings, and over the years has accumulated a set of impressive features. However the peace of mind that your on-disk data is ACID compliant — if not complemented by an equivalent well thought backup strategy — can be easily shattered.
Backup Types
Before diving into the available tools, let’s look at the available PostgreSQL backup types and what their characteristics are:
SQL dumps (or logical)
- Does not block readers or writers.
- Geared towards small sets of data because of the negative impact on system load and the long time required for both backup and restore operations. The performance may be increased with the –no-sync flag, but refer to the man page for the risks associated with disabling the wait for writes.
- A post-restore ANALYZE is required in order to optimize the statistics.
- Global objects such as roles and tablespaces can only be backed up using pg_dumpall utility. Note that tablespace directories must be manually created prior to starting the restore.
- Supports parallelism at the expense of increased system load. Read man pg_dump for its caveats and special requirements e.g. synchronized snapshots.
- Dumps can be loaded in newer versions of PostgreSQL, or even another machine architecture, however they are not guaranteed to be backwards compatible between major versions so some manual editing of the dump file may be required.
Filesystem (or physical)
- Requires the database to be shut down.
- Faster than logical backups.
- Includes cluster data.
- Can only be restored on the same major version of PostgreSQL.
Continuous archiving (or Point In Time Recovery or PITR)
- Suitable for very large databases where logical or physical backups would take too long.
- Some directories inside the data directory can be excluded to speed up the process.
Snapshots
- Requires operating system support — for example LVM works quite well which is also confirmed by NetBackup for PostgreSQL Agent.
- Suitable for applications where both data directory and the database must be in sync e.g. LAMP applications, provided that the two snapshots are synchronized.
- Not recommended when the database files are stored across multiple filesystems (must snapshot all filesystems simultaneously).
Cloud
All cloud providers implement backups in their PostgreSQL offering. Logical backups can be performed as usual, while physical backups and PITR are available through the cloud service offerings since access to the data store is not available (see for example Amazon Aurora for PostgreSQL). Therefore, backing up PostgreSQL in the cloud will need to be a topic for another blog.
Agent base
- Requires an agent installed on targets.
- Can do block-level backups e.g. COMMVAULT (installation supported on Windows only).
Features
While PostgreSQL provides out of the box the tools required to perform logical, physical, and PITR backups, specialized backup applications rely on the native PostgreSQL and operating system tools to fill the need of implementing a backup strategy that addresses the following points:
- automation
- frequency
- retention period
- integrity
- ease of use
Additionally, PostgreSQL backup tools attempt to provide features common to generic backup tools such as:
- incremental backups for saving storage space
- backup catalogs
- ability to store backups on premise or in the cloud
- alerting and notification
- comprehensive reporting
- access control
- encryption
- graphical interface and dashboards
- backups of remote hosts
- adaptive throughput in order to minimize load on the targets
- handling multiple hosts in parallel
- backup orchestration e.g. jobs chaining
- REST APIs
Lab Setup
For this exercise I’ve setup a command-and-control host that where I’ll be installing the backup tools, that also runs two PostgreSQL instances — 9.6 and 10 — installed from PGDG repositories:
[root@cc ~]# ps -o user,pid,ppid,args --forest -U postgres
USER PID PPID COMMAND
postgres 4535 1 /usr/pgsql-10/bin/postmaster -D /var/lib/pgsql/10/data/
postgres 4538 4535 _ postgres: logger process
postgres 4540 4535 _ postgres: checkpointer process
postgres 4541 4535 _ postgres: writer process
postgres 4542 4535 _ postgres: wal writer process
postgres 4543 4535 _ postgres: autovacuum launcher process
postgres 4544 4535 _ postgres: stats collector process
postgres 4545 4535 _ postgres: bgworker: logical replication launcher
postgres 4481 1 /usr/pgsql-9.6/bin/postmaster -D /var/lib/pgsql/9.6/data/
postgres 4483 4481 _ postgres: logger process
postgres 4485 4481 _ postgres: checkpointer process
postgres 4486 4481 _ postgres: writer process
postgres 4487 4481 _ postgres: wal writer process
postgres 4488 4481 _ postgres: autovacuum launcher process
postgres 4489 4481 _ postgres: stats collector process
[root@cc ~]# netstat -npeelt | grep :543
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 26 79972 4481/postmaster
tcp 0 0 127.0.0.1:5433 0.0.0.0:* LISTEN 26 81801 4535/postmaster
tcp6 0 0 ::1:5432 :::* LISTEN 26 79971 4481/postmaster
tcp6 0 0 ::1:5433 :::* LISTEN 26 81800 4535/postmaster
I’ve also setup two remote PostgreSQL instances running the same versions 9.6 and respectively 10:
[root@db-1 ~]# ps -o user,pid,ppid,args --forest -U postgres
USER PID PPID COMMAND
postgres 10972 1 /usr/pgsql-9.6/bin/postmaster -D /var/lib/pgsql/9.6/data/
postgres 10975 10972 _ postgres: logger process
postgres 10977 10972 _ postgres: checkpointer process
postgres 10978 10972 _ postgres: writer process
postgres 10979 10972 _ postgres: wal writer process
postgres 10980 10972 _ postgres: autovacuum launcher process
postgres 10981 10972 _ postgres: stats collector process
[root@db-1 ~]# netstat -npeelt | grep :5432
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN 26 34864 10972/postmaster
tcp6 0 0 :::5432 :::* LISTEN 26 34865 10972/postmaster
[root@db-2 ~]# ps -o user,pid,ppid,args --forest -U postgres
USER PID PPID COMMAND
postgres 10829 1 /usr/pgsql-10/bin/postmaster -D /var/lib/pgsql/10/data/
postgres 10831 10829 _ postgres: logger process
postgres 10833 10829 _ postgres: checkpointer process
postgres 10834 10829 _ postgres: writer process
postgres 10835 10829 _ postgres: wal writer process
postgres 10836 10829 _ postgres: autovacuum launcher process
postgres 10837 10829 _ postgres: stats collector process
postgres 10838 10829 _ postgres: bgworker: logical replication launcher
[root@db-2 ~]# netstat -npeelt | grep :5432
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN 26 34242 10829/postmaster
tcp6 0 0 :::5432 :::* LISTEN 26 34243 10829/postmaster
Next, use pgbench to create a data set:
pgbench=# dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------------+-------+----------+---------+-------------
public | pgbench_accounts | table | postgres | 128 MB |
public | pgbench_branches | table | postgres | 40 kB |
public | pgbench_history | table | postgres | 0 bytes |
public | pgbench_tellers | table | postgres | 40 kB |
(4 rows)
Tools
A list of common backup tools can be found in the PostgreSQL Wiki — Backup section. I’ve augmented the list with products I’ve come across over the years and from a recent Internet search.
Amanda
Amanda is agent based, open source, and supports PostgreSQL out of the box via the ampgsql API. As of this writing, the version 3.5.1 does not support tablespaces (see man ampgsql).
Zmanda provides an enterprise version which is also open source, however not directly available for download as a trial.
Amanda requires a dedicated backup host as the server and client packages exclude each other:
[root@cc ~]# rpm -qp --conflicts ./amanda-backup_client-3.5.1-1.rhel7.x86_64.rpm
amanda-backup_server
[root@cc ~]# rpm -qp --conflicts ./amanda-backup_server-3.5.1-1.rhel7.x86_64.rpm
amanda-backup_client
Follow the basic configuration guide to setup the server and client then configure the PostgreSQL API.
Here’s a git diff from my lab:
- Server:
- increase the server backup space:
--- a/etc/amanda/omiday/amanda.conf +++ b/etc/amanda/omiday/amanda.conf @@ -13,7 +13,7 @@ amrecover_changer "changer" tapetype "TEST-TAPE" define tapetype TEST-TAPE { 1. length 100 mbytes 2. length 500 mbytes filemark 4 kbytes }
- define the PostgreSQL target (and disable sample backup):
--- a/etc/amanda/omiday/disklist +++ b/etc/amanda/omiday/disklist @@ -1,3 +1,2 @@ -localhost /etc simple-gnutar-local +#localhost /etc simple-gnutar-local +10.1.9.243 /var/lib/pgsql/9.6/data dt_ampgsql
- define the PostgreSQL target (and disable sample backup):
- increase the server backup space:
- Client:
- config:
--- /dev/null +++ b/etc/amanda/omiday/amanda-client.conf @@ -0,0 +1,5 @@ +property "PG-DATADIR" "/var/lib/pgsql/9.6/data" +property "PG-ARCHIVEDIR" "/var/lib/pgsql/9.6/archive" +property "PG-HOST" "/tmp" +property "PG-USER" "amandabackup" +property "PG-PASSFILE" "/etc/amanda/pg_passfile"
- authentication file:
--- /dev/null +++ b/etc/amanda/pg_passfile @@ -0,0 +1 @@ +/tmp:*:*:amandabackup:pass
- authentication file:
- authorize the server:
--- a/var/lib/amanda/.amandahosts +++ b/var/lib/amanda/.amandahosts @@ -1,2 +1,3 @@ localhost amandabackup amdump localhost.localdomain amandabackup amdump +10.1.9.231 amandabackup amdump
- PostgreSQL authentication:
--- a/var/lib/pgsql/9.6/data/pg_hba.conf +++ b/var/lib/pgsql/9.6/data/pg_hba.conf @@ -79,7 +79,8 @@ # "local" is for Unix domain socket connections only local all all trust # IPv4 local connections: -host all all 127.0.0.1/32 ident +host all all 127.0.0.1/32 trust +host all amandabackup 10.1.9.243/32 trust # IPv6 local connections: host all all ::1/128 ident # Allow replication connections from localhost, by a user with the
- PostgreSQL config:
--- a/var/lib/pgsql/9.6/data/postgresql.conf +++ b/var/lib/pgsql/9.6/data/postgresql.conf @@ -178,6 +178,7 @@ dynamic_shared_memory_type = posix # the default is the first option #wal_level = minimal # minimal, replica, or logical # (change requires restart) +wal_level = replica #fsync = on # flush data to disk for crash safety # (turning this off can cause # unrecoverable data corruption) @@ -215,10 +216,12 @@ dynamic_shared_memory_type = posix # the default is the first option #archive_mode = off # enables archiving; off, on, or always # (change requires restart) +archive_mode = on #archive_command = '' # command to use to archive a logfile segment # placeholders: %p = path of file to archive # %f = file name only # e.g. 'test ! -f /mnt/server/archivedir/%f && cp %p /mnt/server/archivedir/%f' +archive_command = 'test ! -f /var/lib/pgsql/9.6/archive/%f && cp %p /var/lib/pgsql/9.6/archive/%f' #archive_timeout = 0 # force a logfile segment switch after this # number of seconds; 0 disables
- config:
Once completed the above configuration, run the backup:
[amandabackup@cc ~]$ amdump omiday
And verify:
[amandabackup@cc ~]$ amreport omiday
Hostname: cc
Org : omiday
Config : omiday
Date : April 14, 2018
These dumps were to tape MyData01.
The next tape Amanda expects to use is: MyData02.
STATISTICS:
Total Full Incr. Level:#
-------- -------- -------- --------
Estimate Time (hrs:min) 0:00
Run Time (hrs:min) 0:00
Dump Time (hrs:min) 0:00 0:00 0:00
Output Size (meg) 0.1 0.0 0.1
Original Size (meg) 16.0 0.0 16.0
Avg Compressed Size (%) 0.5 -- 0.5
DLEs Dumped 1 0 1 1:1
Avg Dump Rate (k/s) 33.7 -- 33.7
Tape Time (hrs:min) 0:00 0:00 0:00
Tape Size (meg) 0.1 0.0 0.1
Tape Used (%) 0.0 0.0 0.0
DLEs Taped 1 0 1 1:1
Parts Taped 1 0 1 1:1
Avg Tp Write Rate (k/s) 830.0 -- 830.0
USAGE BY TAPE:
Label Time Size % DLEs Parts
MyData01 0:00 83K 0.0 1 1
NOTES:
planner: tapecycle (3) <= runspercycle (3)
planner: Last full dump of 10.1.9.243:/var/lib/pgsql/9.6/data on tape MyData04 overwritten in 3 runs.
taper: tape MyData01 kb 83 fm 1 [OK]
DUMP SUMMARY:
DUMPER STATS TAPER STATS
HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s
-------------------------------------- ---------------------- -------------- -------------
10.1.9.243 /var/lib/pgsql/9.6/data 1 16416 83 0.5 0:02 33.7 0:00 830.0
(brought to you by Amanda version 3.5.1)
Restoring from backup involves more manual steps as explained in the restore section.
According to the Amanda Enterprise FAQ the following enhancement would apply to our PostgreSQL example:
- management console for automation of backup, retention policies, and schedules
- backup to Amazon S3 cloud storage
Barman
Barman is a disaster recovery solution for PostgreSQL maintained by 2ndQuadrant. It is designed to manage backups for multiple databases and has the ability to restore to a previous point in time using the PITR feature of PostgreSQL.
Barman’s features at a glance:
- handles multiple targets
- support for different PostgreSQL versions
- zero data loss
- streaming and/or standard archiving of WALs
- local or remote recovery
- simplified point in time recovery
As noted in the Barman Manual, support for incremental backups, parallel jobs, data deduplication, and network compression is available only when using the rsync option. Also, streaming WALs from a standby using the archive_command isn’t currently supported.
After following the instructions in the manual for setting up the environment we can verify:
-bash-4.2$ barman list-server
db1 - master
db2 - replica
-bash-4.2$ barman check db1
Server db1:
PostgreSQL: OK
is_superuser: OK
PostgreSQL streaming: OK
wal_level: OK
replication slot: OK
directories: OK
retention policy settings: OK
backup maximum age: OK (no last_backup_maximum_age provided)
compression settings: OK
failed backups: OK (there are 0 failed backups)
minimum redundancy requirements: OK (have 0 backups, expected at least 0)
pg_basebackup: OK
pg_basebackup compatible: OK
pg_basebackup supports tablespaces mapping: OK
archive_mode: OK
archive_command: OK
continuous archiving: OK
pg_receivexlog: OK
pg_receivexlog compatible: OK
receive-wal running: OK
archiver errors: OK
-bash-4.2$ barman check db2
Server db2:
PostgreSQL: OK
is_superuser: OK
PostgreSQL streaming: OK
wal_level: OK
replication slot: OK
directories: OK
retention policy settings: OK
backup maximum age: OK (no last_backup_maximum_age provided)
compression settings: OK
failed backups: OK (there are 0 failed backups)
minimum redundancy requirements: OK (have 0 backups, expected at least 0)
pg_basebackup: OK
pg_basebackup compatible: OK
pg_basebackup supports tablespaces mapping: OK
archive_mode: OK
archive_command: OK
continuous archiving: OK
pg_receivexlog: OK
pg_receivexlog compatible: OK
receive-wal running: OK
archiver errors: OK
Everything checks OK, so we can test by backing up the two hosts:
-bash-4.2$ barman backup db1
Starting backup using postgres method for server db1 in /var/lib/barman/db1/base/20180414T091155
Backup start at LSN: 0/240001B0 (000000010000000000000024, 000001B0)
Starting backup copy via pg_basebackup for 20180414T091155
Copy done (time: 2 seconds)
Finalising the backup.
This is the first backup for server db1
WAL segments preceding the current backup have been found:
000000010000000000000023 from server db1 has been removed
Backup size: 201.9 MiB
Backup end at LSN: 0/26000000 (000000010000000000000025, 00000000)
Backup completed (start time: 2018-04-14 09:11:55.783708, elapsed time: 2 seconds)
Processing xlog segments from file archival for db1
000000010000000000000023
000000010000000000000024
000000010000000000000025.00000028.backup
Processing xlog segments from streaming for db1
000000010000000000000024
-bash-4.2$ barman backup db2
Starting backup using postgres method for server db2 in /var/lib/barman/db2/base/20180414T091225
Backup start at LSN: 0/B0000D0 (00000001000000000000000B, 000000D0)
Starting backup copy via pg_basebackup for 20180414T091225
Copy done (time: 3 seconds)
Finalising the backup.
This is the first backup for server db2
WAL segments preceding the current backup have been found:
000000010000000000000009 from server db2 has been removed
00000001000000000000000A from server db2 has been removed
Backup size: 196.8 MiB
Backup end at LSN: 0/D000000 (00000001000000000000000C, 00000000)
Backup completed (start time: 2018-04-14 09:12:25.619005, elapsed time: 3 seconds)
Processing xlog segments from file archival for db2
00000001000000000000000B
00000001000000000000000C.00000028.backup
Processing xlog segments from streaming for db2
00000001000000000000000B
List the backup catalog:
-bash-4.2$ barman list-backup all
db1 20180414T091155 - Sat Apr 14 09:11:58 2018 - Size: 217.9 MiB - WAL Size: 0 B
db2 20180414T091225 - Sat Apr 14 09:12:28 2018 - Size: 212.8 MiB - WAL Size: 0 B
Displaying the contents for a particular backup:
-bash-4.2$ barman list-files db1 20180414T091155 | head
/var/lib/barman/db1/base/20180414T091155/backup.info
/var/lib/barman/db1/base/20180414T091155/data/backup_label
/var/lib/barman/db1/base/20180414T091155/data/PG_VERSION
/var/lib/barman/db1/base/20180414T091155/data/postgresql.auto.conf
/var/lib/barman/db1/base/20180414T091155/data/pg_ident.conf
/var/lib/barman/db1/base/20180414T091155/data/postgresql.conf
/var/lib/barman/db1/base/20180414T091155/data/pg_hba.conf
When Barman was configured for synchronous WAL streaming we can verify the replication status:
-bash-4.2$ barman replication-status db1
Status of streaming clients for server 'db1':
Current LSN on master: 0/26000528
Number of streaming clients: 1
1. Async WAL streamer
Application name: barman_receive_wal
Sync stage : 3/3 Remote write
Communication : TCP/IP
IP Address : 10.1.9.231 / Port: 37278 / Host: -
User name : streaming_barman
Current state : streaming (async)
Replication slot: barman
WAL sender PID : 2046
Started at : 2018-04-14 09:04:03.019323+00:00
Sent LSN : 0/26000528 (diff: 0 B)
Write LSN : 0/26000528 (diff: 0 B)
Flush LSN : 0/26000000 (diff: -1.3 KiB)
Further enhancements can be added using the provided hook scripts.
Finally, for command line lovers, Barman comes with full TAB completion.
EDB Backup and Recovery Tool (BART)
EDB BART is a closed source proprietary application provided by EnterpriseDB. It combines the PostgreSQL native Filesystem Level Backup and PITR into an easy to use tool providing the following features:
- retention policies
- incremental backups
- complete, hot, physical backups of multiple Postgres Plus Advanced Server and PostgreSQL database servers
- backup and recovery management of the database servers on local or remote hosts
- centralized catalog for backup data
- store backup data in compressed format
- checksum verification
While the trial version for the latest version v2.1 can only be obtained via a yum repo request, the article Data Backup Made Easy and the product documentation guide offer some information for those curious to learn more.
pgBackRest
pgBackRest implements a full system backup that doesn’t rely on the common tools tar and rsync. It is currently hosted and made available by CrunchyData under an MIT license. See Recognition for details on its origins.
It offers all the features one would expect from a PostgreSQL centric tool:
- high backup/restore throughput
- full, incremental, and differential backups
- retention policies
- backup and restore integrity verification through file checksums and integration with PostgreSQL page checksums.
- ability to resume backups
- streaming compression and checksums
- Amazon S3 cloud storage support
- Encryption
..and much more. Refer to the project page for details.
The installation requires a 64-bit Linux/Unix system and it is outlined in the user guide. The guide also introduces the reader to the main concepts, very useful to those new to PostgreSQL or storage technology.
Although the guide uses command examples for Debian/Ubuntu the pgBackRest is available in the PGDG yum repository, and the installer will pull in all the dependencies:
Installing:
pgbackrest x86_64 2.01-1.rhel7 pgdg10 36k
Installing for dependencies:
perl-DBD-Pg x86_64 2.19.3-4.el7 base 195k
perl-DBI x86_64 1.627-4.el7 base 802k
perl-Digest-SHA x86_64 1:5.85-4.el7 base 58k
perl-JSON-PP noarch 2.27202-2.el7 base 55k
perl-Net-Daemon noarch 0.48-5.el7 base 51k
perl-PlRPC noarch 0.2020-14.el7 base 36k
perl-XML-LibXML x86_64 1:2.0018-5.el7 base 373k
perl-version x86_64 3:0.99.07-2.el7 base 84k
Let’s setup two clusters, pg96 and pg10, each having one node:
- control node (“repository” in the guide):
[root@cc ~]# cat /etc/pgbackrest.conf [global] repo1-path=/var/lib/pgbackrest repo1-retention-full=2 start-fast=y [pg96] pg1-path=/var/lib/pgsql/9.6/data pg1-host=db1 pg1-host-user=postgres [pg10] pg1-path=/var/lib/pgsql/10/data pg1-host=db2 pg1-host-user=postgres
- cluster #1:
[root@db-1 ~]# cat /etc/pgbackrest.conf [global] log-level-file=detail repo1-host=repository [pg96] pg1-path=/var/lib/pgsql/9.6/data
- cluster #2:
[root@db-2 ~]# cat /etc/pgbackrest.conf [global] log-level-file=detail repo1-host=repository [pg10] pg1-path=/var/lib/pgsql/10/data
Next, run backups and display the backup catalog:
-bash-4.2$ pgbackrest --stanza=pg96 info
stanza: pg96
status: ok
db (current)
wal archive min/max (9.6-1): 00000001000000000000003D / 00000001000000000000003D
full backup: 20180414-120727F
timestamp start/stop: 2018-04-14 12:07:27 / 2018-04-14 12:08:01
wal start/stop: 00000001000000000000003D / 00000001000000000000003D
database size: 185.6MB, backup size: 185.6MB
repository size: 12.1MB, repository backup size: 12.1MB
-bash-4.2$ pgbackrest --stanza=pg10 info
stanza: pg10
status: ok
db (current)
wal archive min/max (10-1): 000000010000000000000012 / 000000010000000000000012
full backup: 20180414-120810F
timestamp start/stop: 2018-04-14 12:08:10 / 2018-04-14 12:08:38
wal start/stop: 000000010000000000000012 / 000000010000000000000012
database size: 180.5MB, backup size: 180.5MB
repository size: 11.6MB, repository backup size: 11.6MB
pgBackRest supports parallelizing of backup and restore — following the example in the guide, we are backing with one CPU and then update the config to use 2 CPUs:
--- a/etc/pgbackrest.conf
+++ b/etc/pgbackrest.conf
@@ -2,6 +2,7 @@
repo1-path=/var/lib/pgbackrest
repo1-retention-full=2
start-fast=y
+process-max=2
[pg96]
pg1-host=db1
The result:
-bash-4.2$ pgbackrest --stanza=pg96 info
stanza: pg96
status: ok
db (current)
wal archive min/max (9.6-1): 00000001000000000000003D / 000000010000000000000041
full backup: 20180414-120727F
timestamp start/stop: 2018-04-14 12:07:27 / 2018-04-14 12:08:01
wal start/stop: 00000001000000000000003D / 00000001000000000000003D
database size: 185.6MB, backup size: 185.6MB
repository size: 12.1MB, repository backup size: 12.1MB
incr backup: 20180414-120727F_20180414-121434I
timestamp start/stop: 2018-04-14 12:14:34 / 2018-04-14 12:14:52
wal start/stop: 00000001000000000000003F / 00000001000000000000003F
database size: 185.6MB, backup size: 8.2KB
repository size: 12.1MB, repository backup size: 431B
backup reference list: 20180414-120727F
incr backup: 20180414-120727F_20180414-121853I
timestamp start/stop: 2018-04-14 12:18:53 / 2018-04-14 12:19:08
wal start/stop: 000000010000000000000041 / 000000010000000000000041
database size: 185.6MB, backup size: 8.2KB
repository size: 12.1MB, repository backup size: 429B
backup reference list: 20180414-120727F
With 2 CPUs the backup ran almost 20% faster which can make a big difference when running against a large data set.
Conclusion
PostgreSQL centric backup tools offer, as expected, more options than general purpose tools. Most PostgreSQL backup tools offer the same core functionality, but their implementation introduces limitations that can only be discovered by carefully following the documentation to test drive the product.
In addition, ClusterControl offers an array of backup and restore features that you can use as part of your database management setup.