Encrypt an existing Amazon RDS for PostgreSQL DB instance - AWS Prescriptive Guidance

Encrypt an existing Amazon RDS for PostgreSQL DB instance

Created by Piyush Goyal (AWS), Shobana Raghu (AWS), and Yaser Raja (AWS)

Environment: Production

Technologies: Databases; Security, identity, compliance

AWS services: Amazon RDS; AWS KMS; AWS DMS

Summary

This pattern explains how to encrypt an existing Amazon Relational Database Service (Amazon RDS) for PostgreSQL DB instance in the Amazon Web Services (AWS) Cloud with minimal downtime. This process works for Amazon RDS for MySQL DB instances as well.

You can enable encryption for an Amazon RDS DB instance when you create it, but not after it's created. However, you can add encryption to an unencrypted DB instance by creating a snapshot of your DB instance, and then creating an encrypted copy of that snapshot. You can then restore a DB instance from the encrypted snapshot to get an encrypted copy of your original DB instance. If your project allows for downtime (at least for write transactions) during this activity, this is all you need to do. When the new, encrypted copy of the DB instance becomes available, you can point your applications to the new database. However, if your project doesn’t allow for significant downtime for this activity, you need an alternate approach that helps minimize the downtime. This pattern uses the AWS Database Migration Service (AWS DMS) to migrate and continuously replicate the data so that the cutover to the new, encrypted database can be done with minimal downtime. 

Amazon RDS encrypted DB instances use the industry standard AES-256 encryption algorithm to encrypt your data on the server that hosts your Amazon RDS DB instances. After your data is encrypted, Amazon RDS handles authentication of access and decryption of your data transparently, with minimal impact on performance. You don't need to modify your database client applications to use encryption.

Prerequisites and limitations

Prerequisites

  • An active AWS account

  • An unencrypted Amazon RDS for PostgreSQL DB instance

  • Experience working with (creating, modifying, or stopping) AWS DMS tasks (see Working with AWS DMS tasks in the AWS DMS documentation)

  • Familiarity with AWS Key Management Service (AWS KMS) for encrypting databases (see the AWS KMS documentation)

Limitations

  • You can enable encryption for an Amazon RDS DB instance only when you create it, not after the DB instance is created.

  • Data in unlogged tables will not be restored using snapshots. For more information, review Best practices for working with PostgreSQL.

  • You can't have an encrypted read replica of an unencrypted DB instance or an unencrypted read replica of an encrypted DB instance.

  • You can't restore an unencrypted backup or snapshot to an encrypted DB instance.

  • AWS DMS does not automatically transfers the Sequences therefore additional steps are required to handle this.

For more information, see Limitations of Amazon RDS encrypted DB instances in the Amazon RDS documentation.

Architecture

Source architecture

  • Unencrypted RDS DB instance

Target architecture 

  • Encrypted RDS DB instance

    • The destination RDS DB instance is created by restoring the DB snapshot copy of the source RDS DB instance.

    • An AWS KMS key is used for encryption while restoring the snapshot.

    • An AWS DMS replication task is used to migrate the data.

Process uses AWS DMS to encrypt an existing Amazon RDS for PostgreSQL DB instance to a new DB.

Tools

Tools used to enable encryption:

  • AWS KMS key for encryption – When you create an encrypted DB instance, you can choose a customer managed key or the AWS managed key for Amazon RDS to encrypt your DB instance. If you don't specify the key identifier for a customer managed key, Amazon RDS uses the AWS managed key for your new DB instance. Amazon RDS creates an AWS managed key for Amazon RDS for your AWS account. Your AWS account has a different AWS managed key for Amazon RDS for each AWS Region. For more information about using KMS keys for Amazon RDS encryption, see Encrypting Amazon RDS Resources.

Tools used for ongoing replication:

  • AWS DMS – You can use AWS Database Migration Service (AWS DMS) to replicate changes from the source DB to the target DB. It is important to keep the source and target DB in sync to keep downtime to a minimum. For information about setting up AWS DMS and creating tasks, see the AWS DMS documentation.

Epics

TaskDescriptionSkills required

Check the details for the source PostgreSQL DB instance.

On the Amazon RDS console, choose the source PostgreSQL DB instance. On the Configuration tab, make sure that encryption isn't enabled for the instance. For a screen illustration, see the Additional information section.

DBA

Create the DB snapshot.

Create a DB snapshot of the instance you want to encrypt. The amount of time it takes to create a snapshot depends on the size of your database. For instructions, see Creating a DB snapshot in the Amazon RDS documentation.

DBA

Encrypt the snapshot.

In the Amazon RDS console navigation pane, choose Snapshots, and select the DB snapshot you created. For Actions, choose Copy Snapshot. Provide the destination AWS Region and the name of the DB snapshot copy in the corresponding fields. Select the Enable Encryption checkbox. For Master Key, specify the KMS key identifier to use to encrypt the DB snapshot copy. Choose Copy Snapshot. For more information, see Copying a snapshot in the Amazon RDS documentation.

DBA
TaskDescriptionSkills required

Restore the DB snapshot.

On the Amazon RDS console, choose Snapshots. Choose the encrypted snapshot that you created. For Actions, choose Restore Snapshot. For DB Instance Identifier, provide a unique name for the new DB instance. Review the instance details, and then choose Restore DB Instance. A new, encrypted DB Instance will be created from your snapshot. For more information, see Restoring from a DB snapshot in the Amazon RDS documentation.

DBA

Migrate data by using AWS DMS.

On the AWS DMS console, create an AWS DMS task. For Migration type, choose Migrate existing data and replicate ongoing changes. In Task Settings, for Target table preparation mode, choose Truncate. For more information, see Creating a task in the AWS DMS documentation.

DBA

Enable data validation.

In Task Settings, choose Enable validation. This enables you to compare the source data to the target data to verify that the data was migrated accurately. 

DBA

Disable constraints on the target DB instance.

Disable any triggers and foreign key constraints on the target DB instance, and then start the AWS DMS task. For more information about disabling triggers and foreign key constraints, see the AWS DMS documentation.

DBA

Verify data.

After the full load is complete, verify the data on the target DB instance to see if it matches the source data. For more information, see AWS DMS data validation in the AWS DMS documentation.

DBA
TaskDescriptionSkills required

Stop write operations on the source DB instance.

Stop the write operations on the source DB instance so that application downtime can begin. Verify that AWS DMS has completed the replication for the data in the pipeline. Enable triggers and foreign keys on the target DB instance.

DBA

Update database sequences

If the source database contains any sequence numbers, verify and update the sequences in the target database.

DBA

Configure the application endpoint.

Configure your application connections to use the new Amazon RDS DB instance endpoints. The DB instance is now encrypted.

DBA, Application owner

Related resources

Additional information

Checking the encryption for the source PostgreSQL DB instance:

The Summary page of source PostgreSQL DB instance shows encryption not enabled for storage.

Additional notes for this pattern:

  • Enable replication on PostgreSQL by setting the rds.logical_replication parameter to 1.

Important note: Replication slots retain the write ahead log (WAL) files until the files are externally consumed—for example, by pg_recvlogical; by extract, transform, and load (ETL) jobs; or by AWS DMS. When you set the rds.logical_replication parameter value to 1, AWS DMS sets the wal_level, max_wal_senders, max_replication_slots, and max_connections parameters. If logical replication slots are present but there is no consumer for the WAL files retained by the replication slot, you might see an increase in the transaction log disk usage and a constant decrease in free storage space. For more information and steps to resolve this issue, see the article How can I identify what is causing the "No space left on device” or "DiskFull" error on Amazon RDS for PostgreSQL? in the AWS Support Knowledge Center.

  • Any schema changes that you make to the source DB instance after you create the DB snapshot will not be present on the target DB instance.

  • After you create an encrypted DB instance, you can't change the KMS key used by that DB instance. Be sure to determine your KMS key requirements before you create your encrypted DB instance.

  • You must disable triggers and foreign keys on the target DB instance before you run the AWS DMS task. You can re-enable these when the task is complete.