Encrypt an existing Amazon RDS for PostgreSQL DB instance
Created by Piyush Goyal (AWS), Shobana Raghu (AWS), and Yaser Raja (AWS)
Environment: Production | Technologies: Databases; Security, identity, compliance | AWS services: Amazon RDS; AWS KMS; AWS DMS |
Summary
This pattern explains how to encrypt an existing Amazon Relational Database Service (Amazon RDS) for PostgreSQL DB instance in the Amazon Web Services (AWS) Cloud with minimal downtime. This process works for Amazon RDS for MySQL DB instances as well.
You can enable encryption for an Amazon RDS DB instance when you create it, but not after it's created. However, you can add encryption to an unencrypted DB instance by creating a snapshot of your DB instance, and then creating an encrypted copy of that snapshot. You can then restore a DB instance from the encrypted snapshot to get an encrypted copy of your original DB instance. If your project allows for downtime (at least for write transactions) during this activity, this is all you need to do. When the new, encrypted copy of the DB instance becomes available, you can point your applications to the new database. However, if your project doesn’t allow for significant downtime for this activity, you need an alternate approach that helps minimize the downtime. This pattern uses the AWS Database Migration Service (AWS DMS) to migrate and continuously replicate the data so that the cutover to the new, encrypted database can be done with minimal downtime.
Amazon RDS encrypted DB instances use the industry standard AES-256 encryption algorithm to encrypt your data on the server that hosts your Amazon RDS DB instances. After your data is encrypted, Amazon RDS handles authentication of access and decryption of your data transparently, with minimal impact on performance. You don't need to modify your database client applications to use encryption.
Prerequisites and limitations
Prerequisites
An active AWS account
An unencrypted Amazon RDS for PostgreSQL DB instance
Experience working with (creating, modifying, or stopping) AWS DMS tasks (see Working with AWS DMS tasks in the AWS DMS documentation)
Familiarity with AWS Key Management Service (AWS KMS) for encrypting databases (see the AWS KMS documentation)
Limitations
You can enable encryption for an Amazon RDS DB instance only when you create it, not after the DB instance is created.
Data in unlogged tables
will not be restored using snapshots. For more information, review Best practices for working with PostgreSQL. You can't have an encrypted read replica of an unencrypted DB instance or an unencrypted read replica of an encrypted DB instance.
You can't restore an unencrypted backup or snapshot to an encrypted DB instance.
AWS DMS does not automatically transfers the Sequences therefore additional steps are required to handle this.
For more information, see Limitations of Amazon RDS encrypted DB instances in the Amazon RDS documentation.
Architecture
Source architecture
Unencrypted RDS DB instance
Target architecture
Encrypted RDS DB instance
The destination RDS DB instance is created by restoring the DB snapshot copy of the source RDS DB instance.
An AWS KMS key is used for encryption while restoring the snapshot.
An AWS DMS replication task is used to migrate the data.
Tools
Tools used to enable encryption:
AWS KMS key for encryption – When you create an encrypted DB instance, you can choose a customer managed key or the AWS managed key for Amazon RDS to encrypt your DB instance. If you don't specify the key identifier for a customer managed key, Amazon RDS uses the AWS managed key for your new DB instance. Amazon RDS creates an AWS managed key for Amazon RDS for your AWS account. Your AWS account has a different AWS managed key for Amazon RDS for each AWS Region. For more information about using KMS keys for Amazon RDS encryption, see Encrypting Amazon RDS Resources.
Tools used for ongoing replication:
AWS DMS – You can use AWS Database Migration Service (AWS DMS) to replicate changes from the source DB to the target DB. It is important to keep the source and target DB in sync to keep downtime to a minimum. For information about setting up AWS DMS and creating tasks, see the AWS DMS documentation.
Epics
| Task | Description | Skills required |
|---|---|---|
| Check the details for the source PostgreSQL DB instance. | On the Amazon RDS console, choose the source PostgreSQL DB instance. On the Configuration tab, make sure that encryption isn't enabled for the instance. For a screen illustration, see the Additional information section. | DBA |
| Create the DB snapshot. | Create a DB snapshot of the instance you want to encrypt. The amount of time it takes to create a snapshot depends on the size of your database. For instructions, see Creating a DB snapshot in the Amazon RDS documentation. | DBA |
| Encrypt the snapshot. | In the Amazon RDS console navigation pane, choose Snapshots, and select the DB snapshot you created. For Actions, choose Copy Snapshot. Provide the destination AWS Region and the name of the DB snapshot copy in the corresponding fields. Select the Enable Encryption checkbox. For Master Key, specify the KMS key identifier to use to encrypt the DB snapshot copy. Choose Copy Snapshot. For more information, see Copying a snapshot in the Amazon RDS documentation. | DBA |
| Task | Description | Skills required |
|---|---|---|
| Restore the DB snapshot. | On the Amazon RDS console, choose Snapshots. Choose the encrypted snapshot that you created. For Actions, choose Restore Snapshot. For DB Instance Identifier, provide a unique name for the new DB instance. Review the instance details, and then choose Restore DB Instance. A new, encrypted DB Instance will be created from your snapshot. For more information, see Restoring from a DB snapshot in the Amazon RDS documentation. | DBA |
| Migrate data by using AWS DMS. | On the AWS DMS console, create an AWS DMS task. For Migration type, choose Migrate existing data and replicate ongoing changes. In Task Settings, for Target table preparation mode, choose Truncate. For more information, see Creating a task in the AWS DMS documentation. | DBA |
| Enable data validation. | In Task Settings, choose Enable validation. This enables you to compare the source data to the target data to verify that the data was migrated accurately. | DBA |
| Disable constraints on the target DB instance. | Disable any triggers and foreign key constraints | DBA |
| Verify data. | After the full load is complete, verify the data on the target DB instance to see if it matches the source data. For more information, see AWS DMS data validation in the AWS DMS documentation. | DBA |
| Task | Description | Skills required |
|---|---|---|
| Stop write operations on the source DB instance. | Stop the write operations on the source DB instance so that application downtime can begin. Verify that AWS DMS has completed the replication for the data in the pipeline. Enable triggers and foreign keys on the target DB instance. | DBA |
| Update database sequences | If the source database contains any sequence numbers, verify and update the sequences in the target database. | DBA |
| Configure the application endpoint. | Configure your application connections to use the new Amazon RDS DB instance endpoints. The DB instance is now encrypted. | DBA, Application owner |
Related resources
Additional information
Checking the encryption for the source PostgreSQL DB instance:
Additional notes for this pattern:
Enable replication on PostgreSQL by setting the
rds.logical_replicationparameter to 1.
Important note: Replication slots retain the write ahead log (WAL) files until the files are externally consumed—for example, by pg_recvlogical; by extract, transform, and load (ETL) jobs; or by AWS DMS. When you set the rds.logical_replication parameter value to 1, AWS DMS sets the wal_level, max_wal_senders, max_replication_slots, and max_connections parameters. If logical replication slots are present but there is no consumer for the WAL files retained by the replication slot, you might see an increase in the transaction log disk usage and a constant decrease in free storage space. For more information and steps to resolve this issue, see the article How can I identify what is causing the "No space left on device” or "DiskFull" error on Amazon RDS for PostgreSQL?
Any schema changes that you make to the source DB instance after you create the DB snapshot will not be present on the target DB instance.
After you create an encrypted DB instance, you can't change the KMS key used by that DB instance. Be sure to determine your KMS key requirements before you create your encrypted DB instance.
You must disable triggers and foreign keys on the target DB instance before you run the AWS DMS task. You can re-enable these when the task is complete.
