Menu
AWS Database Migration Service
User Guide (Version API Version 2016-01-01)

Migrating Large Data Stores Using AWS Database Migration Service and AWS Snowball

Large-scale data migrations can include many terabytes of information, and can be slowed by network performance and by the sheer amount of data that has to be moved. AWS DMS can load data onto an AWS Snowball device, transfer that data to AWS, and then load the data to the target AWS data store.

Using AWS DMS and the AWS Schema Conversion Tool (AWS SCT), you migrate your data in two stages. First, you use the AWS SCT to process the data locally and then move that data to the AWS Snowball Edge appliance. AWS Snowball then automatically loads the data into an Amazon S3 bucket. Next, when the data is available on Amazon S3, AWS DMS takes the files and migrates the data to the target data store. If you are using change data capture (CDC), those updates are written to the Amazon S3 bucket and the target data store is constantly updated.

AWS Snowball is an AWS service you can use to transfer data to the cloud at faster-than-network speeds using an AWS-owned appliance. An AWS Snowball Edge device can hold up to 100 TB of data. It uses 256-bit encryption and an industry-standard Trusted Platform Module (TPM) to ensure both security and full chain-of-custody for your data.

Amazon S3 is a storage and retrieval service. To store an object in Amazon S3, you upload the file you want to store to a bucket. When you upload a file, you can set permissions on the object and also on any metadata.

AWS DMS supports the following scenarios:

  • Migration from an on-premises data warehouse to Amazon Redshift. This approach involves a client-side software installation of the AWS Schema Conversion Tool. The tool reads information from the warehouse (the extractor), and then moves data to S3 or Snowball. Then in the AWS Cloud, information is either read from S3 or Snowball and injected into Amazon Redshift.

  • Migration from an on-premises relational database to an Amazon RDS database. This approach again involves a client-side software installation of the AWS Schema Conversion Tool. The tool reads information from a local database that AWS supports. The tool then moves data to S3 or Snowball. When the data is in the AWS Cloud, AWS DMS writes it to a supported database in either Amazon EC2 or Amazon RDS.

Process Overview

The process of using AWS DMS and AWS Snowball involves several steps, and it uses not only AWS DMS and AWS Snowball but also the AWS Schema Conversion Tool (AWS SCT). The sections following this overview provide a step-by-step guide to each of these tasks.

Note

We recommend that you test your migration before you use the AWS Snowball device. To do so, you can set up a task to send data, such as a single table, to an Amazon S3 bucket instead of the AWS Snowball device.


                 AWS Database Migration Service and AWS Snowball process

The migration involves a local task, where AWS SCT moves the data to the AWS Snowball Edge device, an intermediate action where the data is copied from the AWS Snowball Edge device to an S3 bucket. The process then involves a remote task where AWS DMS loads the data from the Amazon S3 bucket to the target data store on AWS.

The following steps need to occur to migrate data from a local data store to an AWS data store using AWS Snowball.

  1. Create an AWS Snowball job using the AWS Snowball console. For more information, see Create an Import Job in the AWS Snowball documentation.

  2. Download and install the AWS SCT application on a local machine. The machine must have network access and be able to access the AWS account to be used for the migration. For more information about the operating systems AWS SCT can be installed on, see Installing and Updating the AWS Schema Conversion Tool .

  3. Install the AWS SCT DMS Agent (DMS Agent) on a local, dedicated Linux machine. We recommend that you do not install the DMS Agent on the same machine that you install the AWS SCT application.

  4. Unlock the AWS Snowball Edge device using the local, dedicated Linux machine where you installed the DMS Agent.

  5. Create a new project in AWS SCT.

  6. Configure the AWS SCT to use the DMS Agent.

  7. Register the DMS Agent with the AWS SCT.

  8. Install the database driver for your source database on the dedicated machine where you installed the DMS Agent.

  9. Create and set permissions for the Amazon S3 bucket to use.

  10. Edit the AWS Service Profile in AWS SCT.

  11. Create Local & DMS Task in SCT.

  12. Run and monitor the Local & DMS Task in SCT.

  13. Run the AWS SCT task and monitor progress in SCT.

Step-by-Step Procedures for Migrating Data Using AWS DMS and AWS Snowball

The following sections provide detailed information on the migration steps.

Step 1: Create an AWS Snowball Job

Create an AWS Snowball job by following the steps outlined in the section Getting Started with AWS Snowball Edge: Your First Job in the AWS Snowball documentation.

Step 2: Download and Install the AWS Schema Conversion Tool (AWS SCT)

Download and install the AWS Schema Conversion Tool using the instructions at Installing and Updating the AWS Schema Conversion Tool in the AWS SCT documentation. Install the AWS SCT on a local machine that has access to AWS. This machine should be a different one than that the one where you plan to install the DMS Agent.

Step 3: Install and Configure the AWS SCT DMS Agent

In this step, you install the DMS Agent on a dedicated machine that has access to AWS and to the machine where AWS SCT was installed.

You can install the DMS Agent on the following Linux platforms:

  • Red Hat Enterprise Linux versions 6.2 through 6.8, 7.0 and 7.1 (64-bit)

  • SUSE Linux version 11.1 (64-bit)

To install the DMS Agent

  1. Copy the RPM file called aws-schema-conversion-tool-dms-agent-2.4.0-R2.x86_64.rpm from the AWS SCT installation directory to a dedicated Linux machine.

  2. Run the following command as root to install the DMS Agent. If you install or upgrade the DMS Agent on SUSE Linux, you must add the --nodeps parameter to the command.

    sudo rpm -i amazonrdsmigrationtool-2.2.1-R1.x86_64.rpm

    The default installation location is /opt/amazon/aws-schema-conversion-tool-dms-agent. To install the DMS Agent in a non-default directory, use --prefix <path to new product dir>.

  3. To verify that the Amazon RDS Migration Server is running, issue the following command.

    ps -ef | grep repctl

    The output of this command should show two processes running.

To configure the DMS Agent, you must provide a password and port number. You use the password in AWS SCT, so keep it handy. The port is the one that the DMS Agent should listen on for AWS SCT connections. You might have to configure your firewall to allow connectivity.

Run the following script to configure the DMS Agent.

sudo /opt/amazon/rdsmigrationtool/configure.sh

Step 4: Unlock the AWS Snowball Edge Device

You should run the commands that unlock and provide credentials to the Snowball Edge device from the machine where you installed the DMS Agent. This way you can be sure that the DMS Agent call connect to the AWS Snowball Edge device. For more information about unlocking the AWS Snowball Edge device, see Unlock the Snowball Edge .

For example, the following command lists the Amazon S3 bucket used by the device.

aws s3 ls s3://<bucket-name> --profile <Snowball Edge profile> --endpoint http://<Snowball IP>:8080 --recursive

Step 5: Create a New AWS SCT Project

Next, you create a new AWS SCT project.

To create a new project in AWS SCT

  1. Start AWS SCT, and choose New Project for File. The New Project dialog box appears.

  2. Add the following project information.

    For This Parameter Do This

    Project Name

    Type a name for your project, which is stored locally on your computer.

    Location

    Type the location for your local project file.

    OLTP

    Choose Transactional Database (OLTP).

    Source DB Engine

    Choose your source data store.

    Target DB Engine

    Choose your target data store.

  3. Choose OK to create your AWS SCT project.

  4. (Optional) Test your connection.

Step 6: Configure the AWS SCT Profile to Work with the DMS Agent

The AWS SCT Profile must be updated to use the DMS Agent.

To update the AWS SCT profile to work with the DMS Agent

  1. Start AWS SCT.

  2. Choose Settings, and then choose Global Settings. Choose AWS Service Profiles.

  3. Choose Add New AWS Service Profile.

    
                             AWS Database Migration Service and AWS Snowball process
  4. Add the following profile information.

    For This Parameter Do This

    Profile Name

    Type a name for your profile.

    AWS Access Key

    Type the AWS access key for the AWS account and AWS Region that you plan to use for the migration.

    AWS Secret Key

    Type the AWS secret key for the AWS account and AWS Region that you plan to use for the migration.

    Region

    Choose the AWS Region for the account you are using. Your DMS replication instance, S3 bucket, and target data store must be in this AWS Region.

    S3 Bucket folder

    Type a name for S3 bucket that you were assigned when you created the AWS Snowball job.

  5. After you have entered the information, choose Test Connection to verify that AWS SCT can connect to the Amazon S3 bucket.

    The OLTP Local & DMS Data Migration section in the pop-up window should show all entries with a status of Pass. If the test fails, the failure is probably because the account you are using does not have the correct privileges to access the Amazon S3 bucket.

  6. If the test passes, choose OK and then OK again to close the window and dialog box.

Step 7: Register the DMS Agent in AWS SCT

Next, you register the DMS Agent in AWS SCT. SCT then tries to connect to the agent, showing status. When the agent is available, the status turns to active.

To register the DMS Agent

  1. Start AWS SCT, choose View, and then choose Database Migration View (DMS).

    
                             View dropdown menu
  2. Choose the Agent tab, and then choose Register. The New Agent Registration dialog box appears.

    
                             New Agent Registration dialog box
  3. Type your information in the New Agent Registration dialog box.

    For This Parameter Do This

    Description

    Type the name of the agent, which is DMS Agent.

    Host Name

    Type the IP address of the machine where you installed the DMS Agent.

    Port

    Type the port number that you used when you configured the DMS Agent.

    Password

    Type the password that you used when you configured the DMS Agent.

  4. Choose Register to register the agent with your AWS SCT project.

Step 8: Install the Source Database Driver for the DMS Agent on the Linux Computer

For the migration to succeed, the DMS Agent must be able to connect to the source database. To make this possible, you install the database driver for your source database. The required driver varies by database.

To restart the DMS Agent after database driver installation, change the working directory to <product_dir>/bin and use the steps listed following for each source database.

cd <product_dir>/bin ./arep.ctl stop ./arep.ctl start
To install on Oracle

Install Oracle Instant Client for Linux (x86-64) version 11.2.0.3.0 or later.

In addition, if not already included in your system, you need to create a symbolic link in the $ORACLE_HOME\lib directory. This link should be called libclntsh.so, and should point to a specific version of this file. For example, on an Oracle 12c client:

lrwxrwxrwx 1 oracle oracle 63 Oct 2 14:16 libclntsh.so -> /u01/app/oracle/home/lib/libclntsh.so.12.1

In addition, the LD_LIBRARY_PATH environment variable should be appended with the Oracle lib directory and added to the site_arep_login.sh script under the lib folder of the installation. Add this script if it doesn't exist.

vi cat <product dir>/bin/site_arep_login.sh
export ORACLE_HOME=/usr/lib/oracle/12.2/client64; export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib
To install on Microsoft SQL Server

Install the Microsoft ODBC Driver

Update the site_arep_login.sh with the following code.

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/microsoft/msodbcsql/lib64/

Simba ODBC Driver

Install the Microsoft ODBC Driver

Edit the simba.sqlserverodbc.ini file as follows

DriverManagerEncoding=UTF-16 ODBCInstLib=libodbcinst.so
To install on SAP Sybase

The SAP Sybase ASE ODBC 64-bit client should be installed

If the installation dir is /opt/sap, update the site_arep_login.sh with

export SYBASE_HOME=/opt/sap export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$SYBASE_HOME/ DataAccess64/ODBC/lib:$SYBASE_HOME/DataAccess/ODBC/ lib:$SYBASE_HOME/OCS-16_0/lib:$SYBASE_HOME/OCS-16_0/ lib3p64:$SYBASE_HOME/OCS-16_0/lib3p

The /etc/odbcinst.ini should include these entries

[Sybase] Driver=/opt/sap/DataAccess64/ODBC/lib/libsybdrvodb.so Description=Sybase ODBC driver
To install on MySQL

Install MySQL Connector/ODBC for Linux, version 5.2.6 or later

Make sure that the /etc/odbcinst.ini file contains an entry for MySQL, as in the following example

[MySQL ODBC 5.2.6 Unicode Driver] Driver = /usr/lib64/libmyodbc5w.so UsageCount = 1
To install on PostgreSQL

Install postgresql94-9.4.4-1PGDG.<OS Version>.x86_64.rpm. This is the package that contains the psql executable.

For example, postgresql94-9.4.4-1PGDG.rhel7.x86_64.rpm is the package required for Red Hat 7.

Install the ODBC driver postgresql94-odbc-09.03.0400-1PGDG.<OS version>.x86_64 or above for Linux, where <OS version> is the OS of the agent machine.

For example, postgresql94-odbc-09.03.0400-1PGDG.rhel7.x86_64 is the client required for Red Hat 7.

Make sure that the /etc/odbcinst.ini file contains an entry for PostgreSQL, as in the following example

[PostgreSQL] Description = PostgreSQL ODBC driver Driver = /usr/pgsql-9.4/lib/psqlodbc.so Setup = /usr/pgsql-9.4/lib/psqlodbcw.so Debug = 0 CommLog = 1 UsageCount = 2

Step 9: Configure AWS SCT to Access the Amazon S3 Bucket

For information on configuring an Amazon S3 bucket, see Working with Amazon S3 Buckets in the Amazon S3 documentation.

Note

To use the resulting Amazon S3 bucket in migration, you must have an AWS DMS replication instance created in the same AWS Region as the S3 bucket. If you haven't already created one, do so by using the AWS DMS Management Console, as described in Step 2: Create a Replication Instance.

Step 10: Creating a Local & DMS Task

Next, you create the task that is the end-to-end migration task. The task includes two subtasks. One subtask migrates data from the source database to the AWS Snowball appliance. The other subtask takes the data that the appliance loads into an Amazon S3 bucket and migrates it to the target database.

To create the end-to-end migration task

  1. Start AWS SCT, choose View, and then choose Database Migration View (Local & DMS).

    
                            View > Database Migration View (Local & DMS)
  2. In the left panel that displays the schema from your source database, choose a schema object to migrate. Open the context (right-click) menu for the object, and then choose Create Local & DMS Task.

    
                             AWS Database Migration Service and AWS Snowball process
  3. Add your task information.

    For This Parameter Do This

    Task Name

    Type a name for the task.

    Agent

    Choose DMS Agent.

    Replication Instance

    Choose the AWS DMS replication instance that you want to use.

    Migration Type

    Choose the type of migration you want.

    Choose Migrate existing data to migrate the contents of the chosen schema. This process is called a full load in AWS DMS.

    Choose Migrate existing data and replicate ongoing changes to migrate the contents of the chosen schema and capture all ongoing changes to the database. This process is called full load and CDC in AWS DMS.

    Target table preparation mode

    Choose the preparation mode you want to use.

    Truncate - Tables are truncated without affecting table metadata.

    Drop tables on target - The tables are dropped and new tables are created in their place.

    Do nothing - Data and metadata of the target tables are not changed.

    IAM role

    Choose the predefined IAM role that has permissions to access the Amazon S3 bucket and the target database. For more information about the permissions required to access an Amazon S3 bucket, see Prerequisites When Using S3 as a Source for AWS DMS.

    Logging

    Choose Enable to have AWS CloudWatch create logs for the migration. You incur charges for this service. For more information about AWS CloudWatch, see How Amazon CloudWatch Works

    Description

    Type a description of the task.

    Use Snowball

    Choose this check box to use Snowball.

    Job Name

    Choose the AWS Snowball job name you created.

    Snowball IP

    Type the IP address of the AWS Snowball appliance.

    Port

    Type the port value for the AWS Snowball appliance.

    Local AWS S3 Access key

    Type the AWS access key for the account you are using for the migration.

    Local AWS S3 Secret key

    Type the AWS secret key for the account you are using for the migration.

  4. Choose Create to create the task.

Step 11: Running and Monitoring the Local & DMS Task in SCT

You can start the Local & DMS Task when all connections to endpoints are successful. This means all connections for the Local task, which includes connections from the DMS Agent to the source database, the staging Amazon S3 bucket, and the AWS Snowball device, as well as the connections for the DMS task, which includes connections from the staging Amazon S3 bucket to the target database on AWS.

You can monitor the DMS Agent logs by choosing Show Log. The log details include agent server (Agent Log) and local running task (Task Log) logs. Because the endpoint connectivity is done by the server (since the local task is not running and there are no task logs), connection issues are listed under the Agent Log tab.


                     Local task completed, waiting for the second task

Step 12: Manage the AWS Snowball Appliance

Once the Snowball appliance is fully loaded, AWS SCT updates the status of the task to show it is 50% complete. Remember that the other part of the task involves AWS DMS taking the data from Amazon S3 to the target data store.

To do so, disconnect the AWS Snowball appliance and ship back to AWS. For more information about returning the AWS Snowball appliance to AWS, see the steps outlined in Getting Started with AWS Snowball Edge: Your First Job in the AWS Snowball documentation. You can use the AWS Snowball console or AWS SCT (show details of the DMS task) to check the status of the appliance and find out when AWS DMS begins to load data to the Amazon S3 bucket.


                     Local task completed, waiting for the second task

After the AWS Snowball appliance arrives at AWS and unloads data to S3 bucket, you can see that the remote (DMS) task starts to run. If the migration type you selected for the task was Migrate existing data, the status for the DMS task will show 100% complete when the data has been transferred from Amazon S3 to the target data store. If you set the a task mode to include ongoing replication, then after full load is complete the task status shows that the task continues to run, while AWS DMS applies ongoing changes.

Limitations When Working with AWS Snowball and AWS Database Migration Service (AWS DMS)

There are some limitations you should be aware of when working with AWS Snowball.

  • The LOB mode limits LOB file size to 32K.

  • If an error occurs during the data migration during the load from the local database to the AWS Snowball Edge device or when AWS DMS loads data from Amazon S3 to the target database, the task will restart if the error is recoverable. If AWS DMS cannot recover from the error, the migration will stop.

  • Every AWS SCT task creates two endpoint connections on AWS DMS. If you create multiple tasks, you could reach a resource limit for the number of endpoints that can be created.