Change data capture using AWS Database Migration Service

Publication date: August 2, 2021 (Diagram history)

This architecture shows how to use AWS Database Migration Service (AWS DMS) for change data capture (CDC) to migrate databases using ongoing replication.

Migrate and replicate

The following steps describe the architecture:

1. Sources for CDC include Amazon Aurora, Amazon DocumentDB, Amazon RDS, MongoDB, MySQL, Oracle, PostgreSQL, and SQL Server.

2. AWS Schema Conversion Tool makes heterogeneous database migrations predictable by automatically converting the source database schema and a majority of code objects to a format compatible with the target database.

3. AWS DMS helps you with one-time data migration of databases and continuous data replication. AWS DMS captures changes on the source database and applies them in a transactionally consistent way to the target.

4. Targets for CDC include Oracle, SQL Server, MySQL, PostgreSQL, MongoDB, Amazon Aurora, Amazon DocumentDB, Amazon RDS, and Amazon Simple Storage Service.

Streaming

This architecture shows how to use AWS Database Migration Service (AWS DMS) to ingest real-time changes from source databases through streaming services.

The following steps describe the architecture:

1. Sources for CDC include Oracle, SQL Server, MySQL, PostgreSQL, MongoDB, Amazon Aurora, Amazon DocumentDB, and Amazon RDS.

2. AWS DMS helps you with one-time data migration of databases and continuous data replication. AWS DMS captures changes on the source database and applies them in a transactionally consistent way to the target.

3. Use AWS DMS to stream data to an Amazon Kinesis data stream and Amazon Managed Streaming for Apache Kafka (Amazon MSK) to collect and process large streams of data.

4. Use purpose-built services for analytics in your data lake or data warehouse.

5. Visualize and consume data using AWS Quick Sight and SageMaker AI Notebooks.

Amazon S3 with ETL for upsert

This architecture shows how to use AWS Database Migration Service (AWS DMS) to create data pipelines with change data capture to build transactional data lakes.

The following steps describe the architecture:

1. Sources for CDC include Oracle, SQL Server, MySQL, PostgreSQL, MongoDB, Amazon Aurora, Amazon DocumentDB, and Amazon RDS.

2. AWS DMS helps you with one-time data migration of databases and continuous data replication. AWS DMS captures changes on the source database and applies them in a transactionally consistent way to the target.

3. The target for change data capture is Amazon S3.

4. Use AWS Glue or Amazon EMR for extract, transform, load (ETL) upsert to Amazon S3 and Amazon Redshift.

Further reading

For additional information, refer to the following resources:

• AWS Architecture Icons

• AWS Architecture Center

• AWS Well-Architected