Common Errors When Migrating PostgreSQL Databases From On-Prem to AWS RDS

Ryan King

In part one, Migrating PostgreSQL Databases From On-Prem to the Cloud Using AWS RDS, we showed you how to perform the migration. In this blog, we will explore some common errors that might pop up during the migration.

BACKUP ERROR: Permission Denied to Backup Schema

/usr/pgsql-10/bin/pg_dump -v source_database -h onpremdbserver.domain.com -p 5432 -U source_appuser -Fd -j 100 -f /dbbackup/10/source_database   --no-owner --no-privileges
pg_dump: last built-in OID is 16383
pg_dump: reading extensions
pg_dump: identifying extension members
pg_dump: reading schemas
pg_dump: reading user-defined tables
pg_dump: [archiver (db)] query failed: ERROR:  permission denied for schema source_schema
pg_dump: [archiver (db)] query was: LOCK TABLE source_schema.table1 IN ACCESS SHARE MODE

RESOLUTION - Ensure source_appuser has the proper access. I.e. in this case, source_appuser is owner

postgres=> \c source_database
psql (10.5, server 10.6)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
You are now connected to database "source_database" as user "postgres".
source_database=> \dn+
                               List of schemas
     Name      |   Owner   |   Access privileges    |      Description       
---------------+-----------+------------------------+------------------------
 public        | postgres | postgres=UC/postgres+| standard public schema
               |           | =UC/postgres          |
 source_schema | postgres |                        |
(2 rows)
 
source_database=> alter schema source_schema owner to source_appuser ;
ALTER SCHEMA
 
source_database=> \dt source_schema.table*
             List of relations
    Schema     |  Name  | Type  |   Owner  
---------------+--------+-------+-----------
 source_schema | table1 | table | postgres
 source_schema | table2 | table | postgres
 source_schema | table3 | table | postgres
(3 rows)
 
source_database=> alter table source_schema.table1 owner to source_appuser ;
ALTER TABLE
source_database=> alter table source_schema.table2 owner to source_appuser ;
ALTER TABLE
source_database=> alter table source_schema.table3 owner to source_appuser ;
ALTER TABLE
source_database=> \dt source_schema.table*
                List of relations
    Schema     |  Name  | Type  |     Owner     
---------------+--------+-------+----------------
 source_schema | table1 | table | source_appuser
 source_schema | table2 | table | source_appuser
 source_schema | table3 | table | source_appuser
(3 rows)

BACKUP ERROR: Server Version Mismatch

# Backup database...
/usr/pgsql-10/bin/pg_dump -v source_database -h onpremdbserver.domain.com -p 5432 -U source_appuser -Fd -j 100 -f /dbbackup/10/source_database   --no-owner --no-privileges
pg_dump: server version: 10.6; pg_dump version: 9.6.9
pg_dump: aborting because of server version mismatch

RESOLUTION - Use correct PostgreSQL binary (i.e. /usr/pgsql-10/bin Instead of /usr/pgsql-9.6/bin)

# Backup database...
/usr/pgsql-10/bin/pg_dump -v source_database -h onpremdbserver.domain.com -p 5432 -U source_appuser -Fd -j 100 -f /dbbackup/10/source_database   --no-owner --no-privileges
pg_dump: last built-in OID is 16383
pg_dump: reading extensions
pg_dump: identifying extension members
pg_dump: reading schemas
pg_dump: reading user-defined tables
pg_dump: reading user-defined functions
pg_dump: reading user-defined types
pg_dump: reading procedural languages
pg_dump: reading user-defined aggregate functions
pg_dump: reading user-defined operators
pg_dump: reading user-defined access methods
pg_dump: reading user-defined operator classes
pg_dump: reading user-defined operator families
pg_dump: reading user-defined text search parsers
pg_dump: reading user-defined text search templates
pg_dump: reading user-defined text search dictionaries
pg_dump: reading user-defined text search configurations
pg_dump: reading user-defined foreign-data wrappers
pg_dump: reading user-defined foreign servers
pg_dump: reading default privileges
pg_dump: reading user-defined collations
pg_dump: reading user-defined conversions
pg_dump: reading type casts
pg_dump: reading transforms
pg_dump: reading table inheritance information
pg_dump: reading event triggers
pg_dump: finding extension tables
pg_dump: finding inheritance relationships
pg_dump: reading column info for interesting tables
pg_dump: finding the columns and types of table "source_schema.table1"
pg_dump: finding the columns and types of table "source_schema.table2"
pg_dump: finding the columns and types of table "source_schema.table3"
pg_dump: finding default expressions of table "source_schema.table3"
pg_dump: flagging inherited columns in subtables
pg_dump: reading indexes
pg_dump: reading extended statistics
pg_dump: reading constraints
pg_dump: reading triggers
pg_dump: reading rewrite rules
pg_dump: reading policies
pg_dump: reading row security enabled for table "source_schema.table1"
pg_dump: reading policies for table "source_schema.table1"
pg_dump: reading row security enabled for table "source_schema.table2"
pg_dump: reading policies for table "source_schema.table2"
pg_dump: reading row security enabled for table "source_schema.table3_id_seq"
pg_dump: reading policies for table "source_schema.table3_id_seq"
pg_dump: reading row security enabled for table "source_schema.table3"
pg_dump: reading policies for table "source_schema.table3"
pg_dump: reading publications
pg_dump: reading publication membership
pg_dump: reading publication membership for table "source_schema.table1"
pg_dump: reading publication membership for table "source_schema.table2"
pg_dump: reading publication membership for table "source_schema.table3"
pg_dump: reading subscriptions
pg_dump: reading large objects
pg_dump: reading dependency data
pg_dump: saving encoding = UTF8
pg_dump: saving standard_conforming_strings = on
pg_dump: saving search_path =
pg_dump: saving database definition
pg_dump: dumping contents of table "source_schema.table1"
pg_dump: finished item 3797 TABLE DATA table1
pg_dump: dumping contents of table "source_schema.table3"
pg_dump: finished item 3800 TABLE DATA table3
pg_dump: dumping contents of table "source_schema.table2"
pg_dump: finished item 3798 TABLE DATA table2

RESTORATION ERROR: Must Be the Owner of Extension plpgsql

# Restore database...
/usr/pgsql-10/bin/pg_restore -v -d dest_database_newdb -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -U dest_rds_superuser -j 120 -Fd /dbbackup/10/source_database   --no-owner --no-privileges
pg_restore: connecting to database for restore
pg_restore: processing item 3803 ENCODING ENCODING
pg_restore: processing item 3804 STDSTRINGS STDSTRINGS
pg_restore: processing item 3805 SEARCHPATH SEARCHPATH
pg_restore: processing item 3806 DATABASE source_database
pg_restore: processing item 3 SCHEMA public
pg_restore: creating SCHEMA "public"
pg_restore: processing item 3807 COMMENT SCHEMA public
pg_restore: creating COMMENT "SCHEMA public"
pg_restore: processing item 6 SCHEMA source_schema
pg_restore: creating SCHEMA "source_schema"
pg_restore: processing item 1 EXTENSION plpgsql
pg_restore: creating EXTENSION "plpgsql"
pg_restore: processing item 3808 COMMENT EXTENSION plpgsql
pg_restore: creating COMMENT "EXTENSION plpgsql"
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 3808; 0 0 COMMENT EXTENSION plpgsql
pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of extension plpgsql
    Command was: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
 
ERROR:  permission denied to create database

RESOLUTION: RDS does not allow true super user, but the error can be ignored.

RESTORATION ERROR: Permission Denied to Drop Database

ERROR:
postgres=> drop database dest_database_newdb ;
ERROR:  must be owner of database dest_database_newdb

RESOLUTION

postgres=> grant dest_rds_superuser to postgres ;
GRANT ROLE
postgres=> drop database dest_database_newdb ;
DROP DATABASE

RESTORATION ERROR: Permission Denied to Alter Role

# Grant dest_rds_superuser createdb...
/usr/pgsql-10/bin/psql -E -e postgres -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -Udest_rds_superuser -c"alter role dest_rds_superuser createdb"
alter role dest_rds_superuser createdb
ERROR:  permission denied

RESOLUTION

postgres=> grant rds_superuser to dest_rds_superuser ;
GRANT ROLE
 
 # Grant dest_rds_superuser createdb...
 /usr/pgsql-10/bin/psql -E -e postgres -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -Udest_rds_superuser -c"alter role dest_rds_superuser createdb"
 alter role dest_rds_superuser createdb
 ALTER ROLE

RESTORATION ERROR: Permission Denied to Create Database

ERROR:  permission denied to create database

RESOLUTION

RESOLUTION: Roles must have createdb privilege otherwise to avoid to the following error:
postgres=> alter role dest_rds_superuser createdb ;

 

Download the Whitepaper Today
 
PostgreSQL Management & Automation with ClusterControl
Learn about what you need to know to deploy, monitor, manage and scale PostgreSQL

RESTORATION ERROR: Database Already Exists

# Create database...
/usr/pgsql-10/bin/psql -E -e postgres -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -Udest_rds_superuser -c"create database dest_database_newdb"
create database dest_database_newdb
ERROR:  database "dest_database_newdb" already exists
# Restore database...
/usr/pgsql-10/bin/pg_restore -v -d dest_database_newdb -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -U dest_rds_superuser -j 120 -Fd /dbbackup/10/source_database   --no-owner --no-privileges
pg_restore: connecting to database for restore
pg_restore: processing item 3803 ENCODING ENCODING
pg_restore: processing item 3804 STDSTRINGS STDSTRINGS
pg_restore: processing item 3805 SEARCHPATH SEARCHPATH
pg_restore: processing item 3806 DATABASE source_database
pg_restore: processing item 3 SCHEMA public
pg_restore: creating SCHEMA "public"
pg_restore: processing item 3807 COMMENT SCHEMA public
pg_restore: creating COMMENT "SCHEMA public"
pg_restore: processing item 6 SCHEMA source_schema
pg_restore: creating SCHEMA "source_schema"
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 6; 2615 20233 SCHEMA source_schema source_appuser
pg_restore: [archiver (db)] could not execute query: ERROR:  schema "source_schema" already exists
    Command was: CREATE SCHEMA source_schema;
       pg_restore: processing item 1 EXTENSION plpgsql
       pg_restore: creating EXTENSION "plpgsql"
       pg_restore: processing item 3808 COMMENT EXTENSION plpgsql
       pg_restore: creating COMMENT "EXTENSION plpgsql"
       pg_restore: [archiver (db)] Error from TOC entry 3808; 0 0 COMMENT EXTENSION plpgsql
       pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of extension plpgsql
           Command was: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
       pg_restore: processing item 197 TABLE table1
       pg_restore: creating TABLE "source_schema.table1"
       pg_restore: [archiver (db)] Error from TOC entry 197; 1259 20234 TABLE table1 source_appuser
       pg_restore: [archiver (db)] could not execute query: ERROR:  relation "table1" already exists
           Command was: CREATE TABLE source_schema.table1 (
           id integer
       );
       pg_restore: processing item 198 TABLE table2
       pg_restore: creating TABLE "source_schema.table2"
       pg_restore: [archiver (db)] Error from TOC entry 198; 1259 20237 TABLE table2 source_appuser
       pg_restore: [archiver (db)] could not execute query: ERROR:  relation "table2" already exists
           Command was: CREATE TABLE source_schema.table2 (
           id integer
       );
       pg_restore: processing item 200 TABLE table3
       pg_restore: creating TABLE "source_schema.table3"
       pg_restore: [archiver (db)] Error from TOC entry 200; 1259 20242 TABLE table3 source_appuser
       pg_restore: [archiver (db)] could not execute query: ERROR:  relation "table3" already exists
           Command was: CREATE TABLE source_schema.table3 (
           id integer NOT NULL,
           name character varying
       );
       pg_restore: processing item 199 SEQUENCE table3_id_seq
       pg_restore: creating SEQUENCE "source_schema.table3_id_seq"
       pg_restore: [archiver (db)] Error from TOC entry 199; 1259 20240 SEQUENCE table3_id_seq source_appuser
       pg_restore: [archiver (db)] could not execute query: ERROR:  relation "table3_id_seq" already exists
           Command was: CREATE SEQUENCE source_schema.table3_id_seq
           AS integer
           START WITH 1
           INCREMENT BY 1
           NO MINVALUE
           NO MAXVALUE
           CACHE 1;
       pg_restore: processing item 3809 SEQUENCE OWNED BY table3_id_seq
       pg_restore: creating SEQUENCE OWNED BY "source_schema.table3_id_seq"
       pg_restore: processing item 3675 DEFAULT table3 id
       pg_restore: creating DEFAULT "source_schema.table3 id"
       pg_restore: entering main parallel loop
       pg_restore: launching item 3797 TABLE DATA table1
       pg_restore: launching item 3798 TABLE DATA table2
       pg_restore: launching item 3800 TABLE DATA table3
       pg_restore: launching item 3810 SEQUENCE SET table3_id_seq
       pg_restore: pg_restore: executing SEQUENCE SET table3_id_seq
       processing data for table "source_schema.table2"
       pg_restore: finished item 3798 TABLE DATA table2
       pg_restore: finished item 3810 SEQUENCE SET table3_id_seq
       pg_restore: processing data for table "source_schema.table3"
       pg_restore: processing data for table "source_schema.table1"
       pg_restore: finished item 3797 TABLE DATA table1
       pg_restore: finished item 3800 TABLE DATA table3
       pg_restore: finished main parallel loop
       WARNING: errors ignored on restore: 6

RESOLUTION - Ensure you manually remove existing database before executing the script.

# Restore database...
/usr/pgsql-10/bin/pg_restore -v -d dest_database_newdb -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -U dest_rds_superuser -j 120 -Fd /dbbackup/10/source_database   --no-owner --no-privileges
pg_restore: connecting to database for restore
pg_restore: processing item 3803 ENCODING ENCODING
pg_restore: processing item 3804 STDSTRINGS STDSTRINGS
pg_restore: processing item 3805 SEARCHPATH SEARCHPATH
pg_restore: processing item 3806 DATABASE source_database
pg_restore: processing item 3 SCHEMA public
pg_restore: creating SCHEMA "public"
pg_restore: processing item 3807 COMMENT SCHEMA public
pg_restore: creating COMMENT "SCHEMA public"
pg_restore: processing item 6 SCHEMA source_schema
pg_restore: creating SCHEMA "source_schema"
pg_restore: processing item 1 EXTENSION plpgsql
pg_restore: creating EXTENSION "plpgsql"
pg_restore: processing item 3808 COMMENT EXTENSION plpgsql
pg_restore: creating COMMENT "EXTENSION plpgsql"
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 3808; 0 0 COMMENT EXTENSION plpgsql
pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of extension plpgsql
    Command was: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
 
 
 
pg_restore: processing item 197 TABLE table1
pg_restore: creating TABLE "source_schema.table1"
pg_restore: processing item 198 TABLE table2
pg_restore: creating TABLE "source_schema.table2"
pg_restore: processing item 200 TABLE table3
pg_restore: creating TABLE "source_schema.table3"
pg_restore: processing item 199 SEQUENCE table3_id_seq
pg_restore: creating SEQUENCE "source_schema.table3_id_seq"
pg_restore: processing item 3809 SEQUENCE OWNED BY table3_id_seq
pg_restore: creating SEQUENCE OWNED BY "source_schema.table3_id_seq"
pg_restore: processing item 3675 DEFAULT table3 id
pg_restore: creating DEFAULT "source_schema.table3 id"
pg_restore: entering main parallel loop
pg_restore: launching item 3797 TABLE DATA table1
pg_restore: launching item 3798 TABLE DATA table2
pg_restore: launching item 3800 TABLE DATA table3 

RESTORATION ERROR: Schema Already Exists

# Create schema...
/usr/pgsql-10/bin/psql -E -e -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 dest_database_newdb -Udest_rds_superuser -c"create schema source_schema"
create schema source_schema
ERROR:  schema "source_schema" already exists
 
# Restore database...
/usr/pgsql-10/bin/pg_restore -v -d dest_database_newdb -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -U dest_rds_superuser -j 120 -Fd /dbbackup/10/source_database --schema=source_schema  --no-owner --no-privileges
pg_restore: connecting to database for restore
pg_restore: processing item 3803 ENCODING ENCODING
pg_restore: processing item 3804 STDSTRINGS STDSTRINGS
pg_restore: processing item 3805 SEARCHPATH SEARCHPATH
pg_restore: processing item 6 SCHEMA source_schema
pg_restore: processing item 197 TABLE table1
pg_restore: creating TABLE "source_schema.table1"
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 197; 1259 20234 TABLE table1 source_appuser
pg_restore: [archiver (db)] could not execute query: ERROR:  relation "table1" already exists
    Command was: CREATE TABLE source_schema.table1 (
    id integer
);
 
 
 
pg_restore: processing item 198 TABLE table2
pg_restore: creating TABLE "source_schema.table2"
pg_restore: [archiver (db)] Error from TOC entry 198; 1259 20237 TABLE table2 source_appuser
pg_restore: [archiver (db)] could not execute query: ERROR:  relation "table2" already exists
    Command was: CREATE TABLE source_schema.table2 (
    id integer
);
 
  
pg_restore: processing item 200 TABLE table3
pg_restore: creating TABLE "source_schema.table3"
pg_restore: [archiver (db)] Error from TOC entry 200; 1259 20242 TABLE table3 source_appuser
pg_restore: [archiver (db)] could not execute query: ERROR:  relation "table3" already exists
    Command was: CREATE TABLE source_schema.table3 (
    id integer NOT NULL,
    name character varying
);
 
pg_restore: processing item 199 SEQUENCE table3_id_seq
pg_restore: creating SEQUENCE "source_schema.table3_id_seq"
pg_restore: [archiver (db)] Error from TOC entry 199; 1259 20240 SEQUENCE table3_id_seq source_appuser
pg_restore: [archiver (db)] could not execute query: ERROR:  relation "table3_id_seq" already exists
    Command was: CREATE SEQUENCE source_schema.table3_id_seq
    AS integer
    START WITH 1
    INCREMENT BY 1
    NO MINVALUE
    NO MAXVALUE
    CACHE 1;
 
 
pg_restore: processing item 3806 SEQUENCE OWNED BY table3_id_seq
pg_restore: creating SEQUENCE OWNED BY "source_schema.table3_id_seq"
pg_restore: processing item 3675 DEFAULT table3 id
pg_restore: creating DEFAULT "source_schema.table3 id"
pg_restore: entering main parallel loop
pg_restore: launching item 3797 TABLE DATA table1
pg_restore: launching item 3798 TABLE DATA table2
pg_restore: launching item 3800 TABLE DATA table3
pg_restore: launching item 3807 SEQUENCE SET table3_id_seq
pg_restore: pg_restore: processing data for table "source_schema.table2"
processing data for table "source_schema.table1"
pg_restore: executing SEQUENCE SET table3_id_seq
pg_restore: finished item 3797 TABLE DATA table1
pg_restore: finished item 3798 TABLE DATA table2
pg_restore: finished item 3807 SEQUENCE SET table3_id_seq
pg_restore: processing data for table "source_schema.table3"
pg_restore: finished item 3800 TABLE DATA table3
pg_restore: finished main parallel loop
WARNING: errors ignored on restore: 4

RESOLUTION - Either drop or rename existing schema. I like renaming until all is validated:

postgres=> \c dest_database_newdb
psql (10.5, server 10.6)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
You are now connected to database "dest_database_newdb" as user "postgres".
dest_database_newdb=> \dn
          List of schemas
     Name      |       Owner        
---------------+--------------------
 public        | postgres
 source_schema | dest_rds_superuser
(2 rows)
 
dest_database_newdb=> alter schema source_schema rename to source_schema_old;
ALTER SCHEMA
dest_database_newdb=> \dn
            List of schemas
       Name        |       Owner        
-------------------+--------------------
 public            | postgres
 source_schema_old | dest_rds_superuser
(2 rows)
 
 
# Restore database...
/usr/pgsql-10/bin/pg_restore -v -d dest_database_newdb -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -U dest_rds_superuser -j 120 -Fd /dbbackup/10/source_database --schema=source_schema  --no-owner --no-privileges
pg_restore: connecting to database for restore
pg_restore: processing item 3803 ENCODING ENCODING
pg_restore: processing item 3804 STDSTRINGS STDSTRINGS
pg_restore: processing item 3805 SEARCHPATH SEARCHPATH
pg_restore: processing item 6 SCHEMA source_schema
pg_restore: processing item 197 TABLE table1
pg_restore: creating TABLE "source_schema.table1"
pg_restore: processing item 198 TABLE table2
pg_restore: creating TABLE "source_schema.table2"
pg_restore: processing item 200 TABLE table3
pg_restore: creating TABLE "source_schema.table3"
pg_restore: processing item 199 SEQUENCE table3_id_seq
pg_restore: creating SEQUENCE "source_schema.table3_id_seq"
pg_restore: processing item 3806 SEQUENCE OWNED BY table3_id_seq
pg_restore: creating SEQUENCE OWNED BY "source_schema.table3_id_seq"
pg_restore: processing item 3675 DEFAULT table3 id
pg_restore: creating DEFAULT "source_schema.table3 id"
pg_restore: entering main parallel loop
pg_restore: launching item 3797 TABLE DATA table1
pg_restore: launching item 3798 TABLE DATA table2
pg_restore: launching item 3800 TABLE DATA table3
pg_restore: launching item 3807 SEQUENCE SET table3_id_seq
pg_restore: processing data for table "source_schema.table1"
pg_restore: processing data for table "source_schema.table2"
pg_restore: executing SEQUENCE SET table3_id_seq
pg_restore: finished item 3807 SEQUENCE SET table3_id_seq
pg_restore: processing data for table "source_schema.table3"
pg_restore: finished item 3797 TABLE DATA table1
pg_restore: finished item 3798 TABLE DATA table2
pg_restore: finished item 3800 TABLE DATA table3
pg_restore: finished main parallel loop
ClusterControl
The only management system you’ll ever need to take control of your open source database infrastructure.