Using an Amazon Elasticsearch Service cluster as a target for AWS Database Migration Service - AWS Database Migration Service

Using an Amazon Elasticsearch Service cluster as a target for AWS Database Migration Service

You can use AWS DMS to migrate data to Amazon Elasticsearch Service (Amazon ES). Amazon ES is a managed service that makes it easy to deploy, operate, and scale an Amazon ES cluster.

In Amazon ES, you work with indexes and documents. An index is a collection of documents, and a document is a JSON object containing scalar values, arrays, and other objects. Elasticsearch provides a JSON-based query language, so that you can query data in an index and retrieve the corresponding documents.

When AWS DMS creates indexes for a target endpoint for Amazon ES, it creates one index for each table from the source endpoint. The cost for creating an Amazon ES index depends on several factors. These are the number of indexes created, the total amount of data in these indexes, and the small amount of metadata that Elasticsearch stores for each document.

Configure your Amazon ES cluster with compute and storage resources that are appropriate for the scope of your migration. We recommend that you consider the following factors, depending on the replication task you want to use:

  • For a full data load, consider the total amount of data that you want to migrate, and also the speed of the transfer.

  • For replicating ongoing changes, consider the frequency of updates, and your end-to-end latency requirements.

Also, configure the index settings on your Elasticsearch cluster, paying close attention to the document count.

Multithreaded full load task settings

To help increase the speed of the transfer, AWS DMS supports a multithreaded full load to an Amazon ES target cluster. AWS DMS supports this multithreading with task settings that include the following:

  • MaxFullLoadSubTasks – Use this option to indicate the maximum number of source tables to load in parallel. DMS loads each table into its corresponding Amazon ES target index using a dedicated subtask. The default is 8; the maximum value is 49.

  • ParallelLoadThreads – Use this option to specify the number of threads that AWS DMS uses to load each table into its Amazon ES target index. The maximum value for an Amazon ES target is 32. You can ask to have this maximum limit increased.


    If you don't change ParallelLoadThreads from its default (0), AWS DMS transfers a single record at a time. This approach puts undue load on your Amazon ES cluster. Make sure that you set this option to 1 or more.

  • ParallelLoadBufferSize – Use this option to specify the maximum number of records to store in the buffer that the parallel load threads use to load data to the Amazon ES target. The default value is 50. The maximum value is 1,000. Use this setting with ParallelLoadThreads. ParallelLoadBufferSize is valid only when there is more than one thread.

For more information on how DMS loads an Amazon ES cluster using multithreading, see the AWS blog post Scale Amazon Elasticsearch Service for AWS Database Migration Service migrations.

Multithreaded CDC load task settings

You can improve the performance of change data capture (CDC) for an Amazon ES target cluster using task settings to modify the behavior of the PutRecords API call. To do this, you can specify the number of concurrent threads, queues per thread, and the number of records to store in a buffer using ParallelApply* task settings. For example, suppose you want to perform a CDC load and apply 128 threads in parallel. You also want to access 64 queues per thread, with 50 records stored per buffer.


Support for the use of ParallelApply* task settings during CDC to Amazon Elasticsearch Service target endpoints is available in AWS DMS versions 3.4.0 and later.

To promote CDC performance, AWS DMS supports these task settings:

  • ParallelApplyThreads – Specifies the number of concurrent threads that AWS DMS uses during a CDC load to push data records to a Amazon ES target endpoint. The default value is zero (0) and the maximum value is 32.

  • ParallelApplyBufferSize – Specifies the maximum number of records to store in each buffer queue for concurrent threads to push to a Amazon ES target endpoint during a CDC load. The default value is 50 and the maximum value is 1,000. Use this option when ParallelApplyThreads specifies more than one thread.

  • ParallelApplyQueuesPerThread – Specifies the number of queues that each thread accesses to take data records out of queues and generate a batch load for a Amazon ES endpoint during CDC.

When using ParallelApply* task settings, the partition-key-type default is the primary-key of the table, not schema-name.table-name.

Migrating from a relational database table to an Amazon ES index

AWS DMS supports migrating data to Amazon ES's scalar data types. When migrating from a relational database like Oracle or MySQL to Amazon ES, you might want to restructure how you store this data.

AWS DMS supports the following Amazon ES scalar data types:

  • Boolean

  • Date

  • Float

  • Int

  • String

AWS DMS converts data of type Date into type String. You can specify custom mapping to interpret these dates.

AWS DMS doesn't support migration of LOB data types.

Prerequisites for using Amazon Elasticsearch Service as a target for AWS Database Migration Service

Before you begin work with an Amazon ES database as a target for AWS DMS, make sure that you create an AWS Identity and Access Management (IAM) role. This role should let AWS DMS access the Amazon ES indexes at the target endpoint. The minimum set of access permissions is shown in the following IAM policy.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "1", "Effect": "Allow", "Principal": { "Service": "" }, "Action": "sts:AssumeRole" } ] }

The role that you use for the migration to Amazon ES must have the following permissions.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "es:ESHttpDelete", "es:ESHttpGet", "es:ESHttpHead", "es:ESHttpPost", "es:ESHttpPut" ], "Resource": "arn:aws:es:region:account-id:domain/domain-name/*" } ] }

In the preceding example, replace region with the AWS Region identifier, account-id with your AWS account ID, and domain-name with the name of your Amazon Elasticsearch Service domain. An example is arn:aws:es:us-west-2:123456789012:domain/my-es-domain

Extra connection attributes when using Amazon ES as a target for AWS DMS

When you set up your Amazon ES target endpoint, you can specify extra connection attributes. Extra connection attributes are specified by key-value pairs and separated by semicolons.

The following table describes the extra connection attributes available when using an Amazon ES instance as an AWS DMS source.

Attribute name Valid values Default value and description


A positive integer greater than 0 but no larger than 100.

10 – For a full load task, this attribute determines the threshold of errors allowed before the task fails. For example, suppose that there are 1,500 rows at the source endpoint and this parameter is set to 10. Then the task fails if AWS DMS encounters more than 150 errors (10 percent of the row count) when writing to the target endpoint.


A positive integer greater than 0.

300 – If an error occurs at the target endpoint, AWS DMS retries for this many seconds. Otherwise, the task fails.

Limitations when using Amazon Elasticsearch Service as a target for AWS Database Migration Service

The following limitations apply when using Amazon Elasticsearch Service as a target:

  • Amazon ES uses dynamic mapping (auto guess) to determine the data types to use for migrated data.

  • Amazon ES stores each document with a unique ID. The following is an example ID.

    "_id": "D359F8B537F1888BC71FE20B3D79EAE6674BE7ACA9B645B0279C7015F6FF19FD"

    Each document ID is 64 bytes long, so anticipate this as a storage requirement. For example, if you migrate 100,000 rows from an AWS DMS source, the resulting Amazon ES index requires storage for an additional 6,400,000 bytes.

  • With Amazon ES, you can't make updates to the primary key attributes. This restriction is important when using ongoing replication with change data capture (CDC) because it can result in unwanted data in the target. In CDC mode, primary keys are mapped to SHA256 values, which are 32 bytes long. These are converted to human-readable 64-byte strings, and are used as Amazon ES document IDs.

  • If AWS DMS encounters any items that can't be migrated, it writes error messages to Amazon CloudWatch Logs. This behavior differs from that of other AWS DMS target endpoints, which write errors to an exceptions table.

Target data types for Amazon Elasticsearch Service

When AWS DMS migrates data from heterogeneous databases, the service maps data types from the source database to intermediate data types called AWS DMS data types. The service then maps the intermediate data types to the target data types. The following table shows each AWS DMS data type and the data type it maps to in Amazon ES.

AWS DMS data type Amazon ES data type















For additional information about AWS DMS data types, see Data types for AWS Database Migration Service.