AWS Prescriptive Guidance
Patterns

Migrate an on-premises Oracle database to Amazon RDS for PostgreSQL using an Oracle bystander and AWS DMS

R Type :ReArchitect

source :Databases: Relational

target :Amazon RDS for PostgreSQL/Amazon Aurora PostgreSQL

tags :oracle, database, bystander, postgresql, dms, amazon rds, aws sct

Summary

This pattern describes how you can migrate an on-premises Oracle database to Amazon Relational Database Service (Amazon RDS) for PostgreSQL, using AWS Database Migration Service (AWS DMS) to migrate the data, AWS Schema Conversion Tool (AWS SCT) to convert the database schema, and an Oracle bystander to help manage the migration.

This pattern helps users who want to minimize database downtime. In this implementation, the downtime would be restricted to the length of time it takes to create or validate all the foreign keys on the database. 

The pattern uses Amazon Elastic Compute Cloud (Amazon EC2) instances with an Oracle bystander database to help control the stream of data through AWS DMS. You can temporarily pause streaming replication from the on-premises Oracle database to the Oracle bystander to enable AWS DMS to catch up on data validation or to use another data validation tool. The Amazon RDS for PostgreSQL DB instance and the bystander database will have the same data when AWS DMS has finished migrating current changes. 

Assumptions and Prerequisites

Prerequisites

Limitations

  • Database size limit: 64 TB

Architecture

Source technology stack

  • Oracle versions 11g (versions 11.2.0.3. v1 and later) and 12c, for the Enterprise, Standard, Standard One, and Standard Two editions

  • An EC2 instance that holds a bystander for the Oracle database

Target technology stack

  • Amazon RDS for PostgreSQL, PostgreSQL 9.3 and later

Source and target architecture

Tools Used

AWS DMSAWS Database Migration Service (DMS) helps you migrate databases quickly and securely. The source database remains fully operational during the migration, minimizing downtime to applications that rely on the database. AWS DMS can migrate your data to and from the most widely used commercial and open-source databases.

AWS SCTAWS Schema Conversion Tool (SCT) makes heterogeneous database migrations predictable by automatically converting the source database schema and a majority of the database code objects, including views, stored procedures, and functions, to a format compatible with the target database. Objects that cannot be automatically converted are clearly marked so that they can be manually converted to complete the migration. AWS SCT can also scan your application source code for embedded SQL statements and convert them as part of a database schema conversion project. During this process, AWS SCT performs cloud-native code optimization by converting legacy Oracle and SQL Server functions to their AWS equivalents, to help you modernize your applications while migrating your databases. When schema conversion is complete, AWS SCT can help migrate data from a range of data warehouses to Amazon Redshift by using built-in data migration agents. 

Amazon RDSAmazon Relational Database Service (Amazon RDS) makes it easy to set up, operate, and scale a relational database in the cloud. It provides cost-efficient and resizable capacity while automating time-consuming administration tasks such as hardware provisioning, database setup, patching, and backups. 

Epics

Convert the Oracle database schema to PostgreSQL

Tasks

Title Description Skills Predecessor
Set up AWS SCT. Create a new report, and connect to Oracle as the source and PostgreSQL as the target. In Project Settings, go to the SQL Scripting tab. Change the Target SQL Script to Multiple Files. DBA
Convert the Oracle database schema. In the Action tab, choose Generate Report, Convert Schema, and then Save as SQL. DBA
Modify the scripts. For example, you might want to modify the script if a number in the source schema has been converted to numeric format in PostgreSQL, but you want to use BIGINT instead for better performance. DBA

Create and configure the Amazon RDS DB instance

Tasks

Title Description Skills Predecessor
Create the Amazon RDS DB instance. In the correct AWS Region, in the Amazon RDS console, create a new PostgreSQL DB instance. AWS SysAdmin,DBA
Configure DB instance specifications. Specify the DB engine version, DB instance class, Multi-AZ deployment, storage type, and allocated storage. Enter the DB instance identifier, a master user name, and a master password. AWS SysAdmin,DBA
Configure network and security. Specify the virtual private cloud (VPC), subnet group, public accessibility, Availability Zone preference, and security groups. DBA, SysAdmin
Configure database options. Specify the database name, port, parameter group, encryption, and master key. AWS SysAdmin,DBA
Configure backups. Specify the backup retention period, backup window, start time, duration, and whether to copy tags to snapshots. AWS SysAdmin,DBA
Configure monitoring options. Enable or disable enhanced monitoring and performance insights. AWS SysAdmin,DBA
Configure maintenance options. Specify auto minor version upgrade, maintenance window, and start day, time, and duration. AWS SysAdmin,DBA
Run the pre-migration scripts from AWS SCT. On the Amazon RDS instance, run the following scripts: create_database.sql, create_sequence.sql, create_table.sql, create_view.sql, and create_function.sql. AWS SysAdmin,DBA

Configure the Oracle bystander in Amazon EC2

Tasks

Title Description Skills Predecessor
Set up the network for Amazon EC2. Create the new VPC, subnets, internet gateway, route tables, and security groups. AWS SysAdmin
Create the EC2 instance. In the appropriate AWS Region, create a new EC2 instance. Select the Amazon Machine Image (AMI), choose the instance size, and configure instance details: number of instances (1), the VPC and subnet you created in the previous task, auto-assign public IP, and other options. Add storage, configure security groups, and launch. When prompted, create and save a key pair for the next step. AWS SysAdmin
Connect the Oracle source database to the EC2 instance. Copy the IPv4 public IP address and DNS to a text file and connect by using SSH as follows: ssh -i "your_file.pem" ec2-user@<your-IP-address-or-public-DNS>. AWS SysAdmin
Set up the initial host for a bystander in Amazon EC2. Set up SSH keys, bash profile, ORATAB, and symbolic links. Create Oracle directories. AWS SysAdmin, Linux Admin
Set up the database copy for a bystander in Amazon EC2 Use RMAN to create a database copy, enable supplemental logging, and create the standby control file. After copying is complete, place the database in recovery mode. AWS SysAdmin,DBA
Set up Oracle Data Guard. Modify your listener.ora file and start the listener. Set up a new archive destination. Place the bystander in recovery mode, replace temporary files to avoid future corruption, install a crontab if necessary to prevent the archive directory from running out of space, and edit the manage-trclog-files-oracle.cfg file for the source and standby. AWS SysAdmin,DBA
Prep the Oracle database to sync shipping. Add the standby log files and change the recovery mode. Change the log shipping to SYNC AFFIRM on both the source primary and the source standby. Switch logs on primary, confirm via the Amazon EC2 bystander alert log that you are using the standby log files, and confirm that the redo stream is flowing in SYNC. AWS SysAdmin,DBA

Migrate data with AWS DMS

Tasks

Title Description Skills Predecessor
Create a replication instance in AWS DMS. Complete the fields for the name, instance class, VPC (same as the Amazon EC2 instance), Multi-AZ, and public accessibility. Under Advance, specify allocated storage, subnet group, Availability Zone, VPC security groups, and KMS master key. AWS SysAdmin,DBA
Create the source database endpoint. Specify the endpoint name, type, source engine (Oracle), server name (Amazon EC2 private DNS name), port, SSL mode, user name, password, SID, VPC (specify the VPC that has the replication instance), and replication instance. To test the connection, choose Run Test, and then create the endpoint. You can also configure the following advanced settings: maxFileSize and numberDataTypeScale. AWS SysAdmin,DBA
Connect AWS DMS to Amazon RDS for PostgreSQL. Create a migration security group for connections across VPCs. AWS SysAdmin,DBA
Create the target database endpoint. Specify the endpoint name, type, source engine (PostgreSQL), server name (Amazon RDS endpoint), port, SSL mode, user name, password, database name, VPC (specify the VPC that has the replication instance), and replication instance. To test the connection, choose Run Test, and then create the endpoint. You can also configure the following advanced settings: maxFileSize and numberDataTypeScale. AWS SysAdmin,DBA
Create the AWS DMS replication task. Specify the task name, replication instance, source and target endpoints, and replication instance. For migration type, choose "Migrate existing data and replicate ongoing changes." Clear the "Start task on create" checkbox. AWS SysAdmin,DBA
Configure the AWS DMS replication task settings. For target table preparation mode, choose "Do nothing." Stop task after full load completes (to create primary keys). Specify limited or full LOB mode, and enable control tables. Optionally, you can configure the CommitRate advance setting. DBA
Configure table mappings. In the Table mappings section, create an Include rule for all tables in all schemas included in the migration, and then create an Exclude rule. Add three transformation rules to convert the schema, table, and column names to lowercase, and add any other rules needed for this specific migration. DBA
Start the task. Start the replication task. Make sure that the full load is running. Run ALTER SYSTEM SWITCH LOGFILE on the primary Oracle database to kick-start the task. DBA
Run the mid-migration scripts from AWS SCT. In Amazon RDS for PostgreSQL, run the following scripts: create_index.sql and create_constraint.sql. DBA
Restart the task to continue change data capture (CDC). Run VACUUM on the Amazon RDS for PostgreSQL DB instance, and restart the AWS DMS task to apply cached CDC changes. DBA

Cut over to the PostgreSQL database

Tasks

Title Description Skills Predecessor
Review the AWS DMS logs and validation tables for any errors. Check and fix any replication or validation errors. DBA
Stop all Oracle dependencies. Stop all Oracle dependencies, shut down listeners on the Oracle database, and run ALTER SYSTEM SWITCH LOGFILE. Stop the AWS DMS task when it shows no activity. DBA
Run the post-migration scripts from AWS SCT. In Amazon RDS for PostgreSQL, run these scripts: create_foreign_key_constraint.sql and create_triggers.sql. DBA
Complete additional Amazon RDS for PostgreSQL steps. Increment sequences to match Oracle if needed, run VACUUM and ANALYZE, and take a snapshot for compliance. DBA
Open the connections to Amazon RDS for PostgreSQL. Remove the AWS DMS security groups from Amazon RDS for PostgreSQL, add production security groups, and point your applications to the new database. DBA
Clean up AWS DMS objects. Remove the endpoints, replication tasks, replication instances, and the EC2 instance. SysAdmin, DBA

References and Help

References

Contact and help

Migration Pattern Library Support: aws-mpl@amazon.com