Stream data from IBM Db2, SAP, Sybase, and other databases to MongoDB Atlas on AWS

Battulga Purevragchaa and Igor Alekseev, Amazon Web Services

Babu Srinivasan, MongoDB

Summary

This pattern describes the steps for migrating data from IBM Db2 and other databases such as mainframe databases and Sybase to MongoDB Atlas on the AWS Cloud. It uses AWS Glue to help accelerate the data migration to MongoDB Atlas.

The pattern accompanies the guide Migrating to MongoDB Atlas on AWS on the AWS Prescriptive Guidance website. It provides the implementation steps for one of the migration scenarios that are discussed in that guide. For additional migration scenarios, see the following patterns on the AWS Prescriptive Guidance website:

• Migrate a self-hosted MongoDB environment to MongoDB Atlas on AWS

• Migrate relational databases to MongoDB Atlas on AWS

The pattern is intended for AWS Managed Services Partners and AWS users.

Prerequisites and limitations

Prerequisites

• A source database such as SAP, Sybase, IBM Db2, and others to migrate to MongoDB Atlas.

• Familiarity with databases such as SAP, Sybase, IBM Db2, MongoDB Atlas, and AWS services.

Product versions

• MongoDB version 5.0 or later.

Architecture

The following diagram illustrates batch data load and data streaming by using AWS Glue Studio, Amazon Kinesis Data Streams, and MongoDB Atlas.

This reference architecture uses AWS Glue Studio to create extract, transform, and load (ETL) pipelines to migrate data to MongoDB Atlas. An AWS Glue crawler integrates with MongoDB Atlas to facilitate data governance. The data can be either ported in batch or streamed to MongoDB Atlas by using Amazon Kinesis Data Streams.

Batch data load

For more information about the batch data migration, see the AWS blog post Compose your ETL jobs for MongoDB Atlas with AWS Glue.

Data streaming

For MongoDB Atlas reference architectures that support different usage scenarios, see Migrating to MongoDB Atlas on AWS on the AWS Prescriptive Guidance website.

Tools

●      AWS Glue is a fully managed ETL service. It helps you reliably categorize, clean, enrich, and move data between data stores and data streams.

●      Amazon Kinesis Data Streams helps you collect and process large streams of data records in real time.

●      MongoDB Atlas is a fully managed database as a service (DbaaS) for deploying and managing MongoDB databases in the cloud.

Best practices

For guidelines, see Best Practices Guide for MongoDB in the MongoDB GitHub repository.

Epics

Discovery and assessment

Task	Description	Skills required
Determine the cluster size.	Estimate the working set size by using the information from db.stats() for the total index space. Assume that a percentage of your data space will be accessed frequently. Or, you can estimate your memory requirements based on your assumptions. This task should take approximately one week. For more information and examples for this and the other stories in this epic, see the links in the Related resources section.	MongoDB DBA, Application architect
Estimate network bandwidth requirements.	To estimate your network bandwidth requirements, multiply the average document size by the number of documents served per second. Consider the maximum traffic that any node on your cluster will bear as the basis. To calculate downstream data transfer rates from your cluster to client applications, use the sum of the total documents returned over a period of time. If your applications read from secondary nodes, divide this number of total documents by the number of nodes that can serve read operations. To find the average document size for a database, use the db.stats().avgObjSize command. This task will typically take one day.	MongoDB DBA
Select the Atlas tier.	Follow the instructions in the MongoDB documentation to select the correct Atlas cluster tier.	MongoDB DBA
Plan for cutover.	Plan for application cutover.	MongoDB DBA, Application architect

Set up a new MongoDB Atlas environment on AWS

Task	Description	Skills required
Create a new MongoDB Atlas cluster on AWS.	In MongoDB Atlas, choose Build a Cluster, and select AWS as the cloud provider.	MongoDB DBA
Select AWS Regions and global cluster configuration.	Select from the list of available AWS Regions for your Atlas cluster. Configure global clusters if required.	MongoDB DBA
Select the cluster tier.	Select your preferred cluster tier. Your tier selection determines factors such as memory, storage, and IOPS specification.	MongoDB DBA
Configure additional cluster settings.	Configure additional cluster settings such as MongoDB version, backup, and encryption options. For more information about these options, see the Related resources section.	MongoDB DBA

Configure security and compliance

Task	Description	Skills required
Configure the access list.	To connect to the Atlas cluster, you must add an entry to the project’s access list. Atlas uses Transport Layer Security (TLS) / Secure Sockets Layer (SSL) to encrypt the connections to the virtual private cloud (VPC) for your database. To set up the access list for the project and for more information about the stories in this epic, see the links in the Related resources section.	MongoDB DBA
Authenticate and authorize users.	You must create and authenticate the database users who will access the MongoDB Atlas clusters. To access the clusters in a project, users must belong to that project, and they can belong to multiple projects. You can also enable authorization with AWS Identity and Access Management (IAM). For more information, see Set Up Authentication with IAM in the MongoDB documentation.	MongoDB DBA
Create custom roles.	(Optional) Atlas supports creating custom roles if the built-in Atlas database user privileges don’t cover your desired set of privileges.	MongoDB DBA
Set up VPC peering.	(Optional) Atlas supports VPC peering with other AWS VPCs.	MongoDB DBA
Set up an AWS PrivateLink endpoint.	(Optional) You can set up private endpoints on AWS by using AWS PrivateLink.	MongoDB DBA
Enable two-factor authentication.	(Optional) Atlas supports two-factor authentication (2FA) to help users control access to their Atlas accounts.	MongoDB DBA
Set up user authentication and authorization with LDAP.	(Optional) Atlas supports performing user authentication and authorization with Lightweight Directory Access Protocol (LDAP).	MongoDB DBA
Set up unified AWS access.	(Optional) Some Atlas features, including Atlas Data Lake and encryption at rest using customer key management, use IAM roles for authentication.	MongoDB DBA
Set up encryption at rest by using AWS KMS.	(Optional) Atlas supports using AWS Key Management Service (AWS KMS) to encrypt storage engines and cloud provider backups.	MongoDB DBA
Set up CSFLE.	(Optional) Atlas supports client-side field-level encryption (CSFLE), including automatic encryption of fields.	MongoDB DBA

Migrate data

Task	Description	Skills required
Launch your target replica set in MongoDB Atlas.	Launch your target replica set in MongoDB Atlas. In Atlas Live Migration Service, choose I'm ready to migrate.	MongoDB DBA
Establish the connection of AWS Glue with MongoDB Atlas.	Use an AWS Glue crawler to connect AWS Glue with MongoDB Atlas (target database). This step helps prepare the target environment for migration. For more information, see the AWS Glue documentation.	MongoDB DBA
Establish the connection of AWS Glue with the source database or source stream.	This helps prepare the target environment for migration.	MongoDB DBA
Set up the data transformation.	Configure the transformation logic to migrate the data from the legacy structured schema to the flexible schema of MongoDB.	MongoDB DBA
Migrate the data.	Schedule the migration in AWS Glue Studio.	MongoDB DBA

Configure operational integration

Task	Description	Skills required
Connect to the cluster.	Connect to the MongoDB Atlas cluster.	App developer
Interact with data.	Interact with cluster data.	App developer
Monitor the clusters.	Monitor your MongoDB Atlas clusters.	MongoDB DBA
Back up and restore data.	Back up and restore cluster data.	MongoDB DBA

Troubleshooting

Issue	Solution
If you encounter issues	See Troubleshooting in the MongoDB Atlas CloudFormation Resources repository.

Related resources

All of the following links, unless noted otherwise, go to webpages in the MongoDB documentation.

Migration guide

• Migrating to MongoDB Atlas on AWS (AWS Prescriptive Guidance)

Discovery and assessment

• Memory

• Sizing example with Atlas sample data sets

• Sizing example for mobile applications

• Network Traffic

• Cluster Auto-Scaling

• Atlas sizing template

Configuring security and compliance

• Configure IP Access List Entries

• Configure Database Users

• Configure Access to the Atlas UI

• Configure Custom Database Roles

• Configure Database Users

• Set up a Network Peering Connection

• Learn About Private Endpoints in Atlas

• Manage Your Multi-Factor Authentication Options

• Set up User Authentication and Authorization with LDAP

• Atlas Data Lake

• Encryption at Rest using Customer Key Management

• Methods to assume a role (IAM documentation)

• Client-Side Field Level Encryption

• Automatic Encryption 

• MongoDB Atlas Security Controls

• MongoDB Trust Center

• Configure Security Features for Clusters

Setting up a new MongoDB Atlas environment on AWS

• Cloud Providers and Regions

• Manage Global Clusters

• Select Cluster Tier

• Configure Additional Settings

• Get Started with Atlas

• Configure Access to the Atlas UI

• Manage Clusters

Migrating data

• Migrate or Import Data

Monitoring clusters

• Monitor Your Clusters

Integrating operations

• Connect to a Cluster

• Interact with Your Data

• Monitor Your Clusters

• Backu Up, Restore, and Archive Data

GitHub repository

• Stream data to MongoDB Atlas using AWS Glue