Setting up managed disaster recovery in Amazon OpenSearch Service

Amazon OpenSearch Service managed disaster recovery (DR) replicates data from a primary domain in one AWS Region to a secondary domain in another Region. The service manages the connection between the domains and encrypts replicated data in transit, so you don't need to configure network connectivity, such as VPC peering, between the domains.

When you set up DR, the service copies each existing index from the primary domain to the secondary domain. After this initial copy, the service asynchronously replicates new changes from the primary domain, and the secondary domain indexes them.

Important

DR is supported only when both the primary and secondary domains run OpenSearch 3.7 or later. To upgrade a domain, see Upgrading Amazon OpenSearch Service domains.

DR uses an active-passive model. Both domains are available for read operations, but only the primary domain accepts write operations. If the primary domain becomes unavailable, you can manually fail over to the secondary domain. You can also perform a planned switchover to exchange the primary and secondary roles without data loss.

Use Multi-AZ to protect a domain from Availability Zone failures within one AWS Region. Use DR when your recovery plan requires a domain in another Region.

Requirements and limitations

Before you set up DR, review the following requirements and limitations:

• Both domains must run the same OpenSearch version, 3.7 or later. If the versions differ later, for example during an upgrade, you can't perform a switchover until both domains run the same version. For more information, see Upgrade domains that use DR.

• Both domains must be in the same AWS account and in different AWS Regions. A domain can have only one replication peer.

• Both domains must have fine-grained access control and node-to-node encryption enabled.

• The secondary domain must be empty. It can't contain any indexes other than system indexes, because DR copies all indexes from the primary domain to the secondary domain.

• Both domains index all of the data, and each index has the same number of primary shards and replicas on both domains. For consistent performance after a switchover or failover, configure the domains with similar CPU, memory, and storage capacity.

• The secondary domain must have at least as much total storage as the primary domain, and enough free storage for the primary domain's current index data. DR checks storage only when you set up the replication relationship.

• Warm and cold storage aren't supported. Neither domain can use UltraWarm, cold storage, or the writable warm tier of multi-tier storage.

• Domains that use the Optimized engine for log analytics aren't supported.

• Automatic semantic enrichment isn't supported on domains that use DR replication. You can't pair domains when either domain contains an index that has automatic semantic enrichment enabled. Delete these indexes before you create the replication relationship.

  While the relationship exists, you can't enable automatic semantic enrichment for a new or existing index on either domain. To use automatic semantic enrichment again, first remove the replication relationship.

• Failover is manual. OpenSearch Service doesn't determine when to fail over or automatically promote the secondary domain. To automate failover, build your own automation that calls FailoverReplication, for example in response to CloudWatch alarms.

• DR doesn't redirect your applications after a switchover or failover. Update your applications to send writes to the new primary domain.

Data and resources that DR replicates

DR replicates the following data and resources from the primary domain:

• Indexes and documents

• Index settings

• Aliases

• Data streams

DR doesn't replicate the following resources:

• Index templates, including composable index templates

• Ingest pipelines

• Index State Management (ISM) policies

• Roles

• Cluster settings

Important

Before a switchover or failover, create the same index templates, ingest pipelines, ISM policies, and roles on the secondary domain. Without the required resources on the new primary, data stream rollover can fail, new indexes don't inherit the expected settings and mappings, and writes that use an ingest pipeline fail.

Use infrastructure as code to keep the domain configurations and non-replicated resources consistent. You can use the CompareDomainConfiguration API to identify differences in key configuration categories between the domains.

Closed indexes

A closed index isn't protected by DR replication. When you close an index on the primary domain, DR removes its copy from the secondary domain. If you fail over while the index is closed, the index isn't available on the secondary domain.

When you reopen the index, DR copies the entire index to the secondary domain again. The index is protected after this copy finishes. Copying a large index can take a long time.

Set up disaster recovery

Create and configure both domains before you establish the replication relationship. We recommend that you use infrastructure as code so that you can apply consistent domain configurations and non-replicated resources in both Regions.

Console

To set up disaster recovery

1. Open the Amazon OpenSearch Service console at https://console.aws.amazon.com/aos/home.

2. In the navigation pane, choose Domains, and then choose the domain that will be the primary domain.

3. In the General information section, for Disaster recovery, choose Set up.

4. For Secondary domain ARN, enter the ARN of the secondary domain.

5. Choose Validate prerequisites. If the domains are compatible, the console confirms that they can be paired for disaster recovery. Otherwise, the console describes each issue that prevents pairing. Fix the issues, and then validate again. For the requirements, see Requirements and limitations.

6. Choose Create and enable disaster recovery. Creating the replication relationship can take several minutes.

AWS CLI

Call AddReplicationPeer in the Region of the primary domain. The following AWS CLI command adds the secondary domain as a replication peer:

aws opensearch add-replication-peer \
  --domain-name primary-domain-name \
  --peer-domain-arn arn:aws:es:secondary-region:account-id:domain/secondary-domain-name \
  --region primary-region

To validate the domains without creating the replication relationship, add the --dry-run option. OpenSearch Service creates the relationship asynchronously. To check when replication is active, see Check replication status.

Check replication status

After you set up DR, and after each switchover or failover, check the role of each domain and the status of the replication relationship.

Console

To check replication status

1. Open the Amazon OpenSearch Service console at https://console.aws.amazon.com/aos/home.

2. In the navigation pane, choose Domains, and then choose either domain.

3. In the General information section, for Disaster recovery, choose Manage. The Disaster recovery page shows the primary domain, the secondary domain, and the replication status.

AWS CLI

The following AWS CLI command calls DescribeDomainReplication for either domain. Run the command in that domain's Region:

aws opensearch describe-domain-replication \
  --domain-name domain-name \
  --region region

The response includes the role of the domain that you specify and, for its peer, the peer's role and the status of the relationship. The status is ACTIVE when replication is running.

{
  "DomainArn": "arn:aws:es:primary-region:account-id:domain/primary-domain-name",
  "Role": "PRIMARY",
  "Peers": [
    {
      "PeerDomainArn": "arn:aws:es:secondary-region:account-id:domain/secondary-domain-name",
      "PeerRole": "SECONDARY",
      "Status": "ACTIVE"
    }
  ]
}

Remove the replication relationship

When you remove the replication relationship, replication stops for both domains. Both domains become standalone domains and keep their data. To protect a domain again, set up DR with an empty secondary domain.

Console

To remove the replication relationship

1. Open the Amazon OpenSearch Service console at https://console.aws.amazon.com/aos/home.

2. In the navigation pane, choose Domains, and then choose either domain.

3. In the General information section, for Disaster recovery, choose Manage.

4. Choose Actions, and then choose Remove replication peer.

5. Enter remove.

6. Choose Remove peer.

AWS CLI

The following AWS CLI command calls RemoveReplicationPeer. Run the command in the Region of the domain that you specify, and specify the other domain as the peer:

aws opensearch remove-replication-peer \
  --domain-name domain-name \
  --peer-domain-arn arn:aws:es:peer-region:account-id:domain/peer-domain-name \
  --region region

The status is PENDING_REMOVAL while replication is being stopped. After the removal finishes, the relationship no longer appears in the DescribeDomainReplication response.

Disaster recovery API operations

The Amazon OpenSearch Service configuration API provides the following operations for managing DR:

AddReplicationPeer

Adds a secondary domain as a replication peer.

RemoveReplicationPeer

Removes the replication relationship. Each domain retains its data.

SwitchoverReplication

Performs a planned switchover without data loss and exchanges the primary and secondary roles. Call this operation on the secondary domain that you want to promote.

FailoverReplication

Unilaterally promotes a secondary domain to primary. Call this operation on the secondary domain that you want to promote.

DescribeDomainReplication

Describes the relationship, including the role and replication status of each domain.

CompareDomainConfiguration

Compares key configuration categories between two domains and reports the differences.

For API syntax and response details, see the Amazon OpenSearch Service API Reference.

CloudWatch metrics

The secondary domain publishes the following DR metrics to Amazon CloudWatch every minute. The primary domain doesn't publish these metrics. We recommend that you create CloudWatch alarms for these metrics. For information about viewing metrics, see Monitoring OpenSearch cluster metrics with Amazon CloudWatch.

Disaster recovery metrics

Metric	Description	Unit
DisasterRecoveryReplicationLagTime	The approximate amount of time that the secondary domain is behind the primary domain. This metric can report values up to 10 seconds even when the secondary domain is fully caught up.	Milliseconds
DisasterRecoveryReplicationLagOps	The number of changes that the secondary domain hasn't applied. A value of 0 means that the secondary domain is caught up. When the primary domain is receiving writes, this value is usually greater than 0, and its typical value depends on your write rate.	Count
DisasterRecoveryBootstrappingIndices	The number of indexes performing their first full copy from the primary domain. This value can be greater than 0 immediately after setup or after you create a new index. It should return to 0 after the copy finishes.	Count
DisasterRecoveryDegradedIndices	The number of indexes that are replicating but aren't fully healthy. This value should be 0.	Count
DisasterRecoveryFailedIndices	The number of indexes where replication stopped because of an error. These indexes no longer receive changes from the primary domain. This value should be 0.	Count

Perform a planned switchover

Use a planned switchover when both domains are available and you want to move your workload to the secondary Region, for example to rehearse your recovery procedure or to perform maintenance. Before the switchover, verify that both domains run the same engine version, compare the domain configurations, and verify that the secondary domain contains the required non-replicated resources.

OpenSearch Service coordinates with the current primary domain, replicates any pending changes, and exchanges the primary and secondary roles without data loss. During the switchover, neither domain accepts writes for a short time, so configure your applications to retry failed writes. After the switchover, the former primary domain becomes the secondary domain, and replication continues in the reverse direction.

Console

To perform a planned switchover

1. Open the Amazon OpenSearch Service console at https://console.aws.amazon.com/aos/home.

2. In the navigation pane, choose Domains, and then choose the secondary domain.

3. In the General information section, for Disaster recovery, choose Manage.

4. Choose Switchover.

5. Enter confirm.

6. Choose Start switchover.

AWS CLI

The following AWS CLI command calls SwitchoverReplication in the Region of the secondary domain that you want to promote. For the peer, specify the current primary domain:

aws opensearch switchover-replication \
  --domain-name secondary-domain-name \
  --peer-domain-arn arn:aws:es:primary-region:account-id:domain/primary-domain-name \
  --region secondary-region

The status is SWITCHING_OVER while the switchover is in progress. To confirm the new roles, see Check replication status.

Important

After the switchover, redirect application writes to the new primary domain. Amazon OpenSearch Service doesn't update your application configuration or endpoints.

Perform an unplanned failover

Use an unplanned failover when the primary domain is unavailable and you decide to promote the secondary domain. OpenSearch Service promotes the secondary domain without coordinating with the primary domain.

Warning

An unplanned failover can cause data loss. Writes that the old primary domain accepted but didn't replicate before the failover are lost, and the replication lag at the time of the failover limits this loss. If clients keep writing to the old primary domain after the failover, those writes also aren't replicated to the new primary domain.

Console

To perform an unplanned failover

1. Open the Amazon OpenSearch Service console at https://console.aws.amazon.com/aos/home.

2. In the navigation pane, choose Domains, and then choose the secondary domain.

3. In the General information section, for Disaster recovery, choose Manage.

4. Choose Failover.

5. Select the check box to acknowledge that data might be lost.

6. Enter failover.

7. Choose Fail over.

AWS CLI

The following AWS CLI command calls FailoverReplication in the Region of the secondary domain that you want to promote. For the peer, specify the current primary domain. Because a failover can lose data, you must include the --allow-data-loss option:

aws opensearch failover-replication \
  --domain-name secondary-domain-name \
  --peer-domain-arn arn:aws:es:primary-region:account-id:domain/primary-domain-name \
  --allow-data-loss \
  --region secondary-region

The status is FAILING_OVER while the failover is in progress. To confirm the new roles, see Check replication status.

After the failover, redirect application writes to the new primary domain, and stop clients from writing to the former primary domain. Review the per-index replication status before you rely on the new primary for recovery.

The former primary domain doesn't rejoin replication automatically. Its replication status is EJECTED, and DR doesn't protect the new primary domain until you pair it with a secondary domain again. When OpenSearch Service can reach the former primary domain again, the domain becomes read-only. If the former primary domain kept running during the failure, for example during a network partition, clients that could still reach it might have written to it after the failover. Those writes aren't replicated to the new primary domain. For information about saving those writes before you fail back, see Fail back after a failover.

Fail back after a failover

After a failover, DR doesn't resynchronize the former primary domain. To protect the new primary domain, pair it with an empty secondary domain, and DR copies all indexes to that domain. To return the primary role to the original Region, use a domain in the original Region as the new secondary domain, and then perform a planned switchover after the copy finishes.

To fail back after a failover

1. When the original Region is available, remove the replication relationship from the former primary domain. In the console, start from the former primary domain. With the AWS CLI, specify the former primary domain as the domain and the new primary domain as the peer. For instructions, see Remove the replication relationship.

2. Delete all indexes except system indexes from the former primary domain, or create a new domain in the original Region. The secondary domain must be empty when you set up DR.

   Important

   The former primary domain might contain writes that the new primary domain doesn't have. These include writes that weren't replicated before the failover and, if the former primary domain kept running, writes that clients made to it after the failover. Before you delete its indexes, save any data that you need, for example by taking a manual snapshot.

3. Set up DR with the new primary domain as the primary domain and the empty domain in the original Region as the secondary domain. For instructions, see Set up disaster recovery.

4. Wait for the initial copy to finish. The DisasterRecoveryBootstrappingIndices metric returns to 0 after all indexes are copied. Copying a large amount of data can take several hours.

5. (Optional) To make the domain in the original Region the primary domain again, perform a planned switchover. For instructions, see Perform a planned switchover.

Recover from a long secondary-domain outage

The primary domain keeps a limited history of index operations so that the secondary domain can replay changes after an interruption. If the secondary domain is unavailable for longer than this history is retained, DR can't catch up the affected indexes.

The affected indexes on the secondary domain remain searchable, but their results are stale. For example, a search can return an older version of an updated document or a document that was deleted from the primary domain. The overall domain replication status can still be ACTIVE while the per-index replication status identifies the affected indexes as failed.

To recover an affected index, initiate a new full copy, or re-bootstrap, from the secondary domain. The index is protected again after the copy finishes.

Upgrade domains that use DR

When you upgrade the engine version of domains that use DR, upgrade the secondary domain first, and then upgrade the primary domain. You can't perform a switchover while the domains run different versions. The steps depend on the upgrade method that you use. For information about upgrade methods, see Choose an upgrade method.

In-place upgrades

An in-place upgrade upgrades the existing cluster of a domain and runs as a blue/green deployment. In-place upgrades work while DR is active, so you don't need to remove the replication relationship.

To upgrade domains with an in-place upgrade

1. Upgrade the secondary domain to the new version.

2. Validate your workload on the secondary domain. For example, test your read requests against the secondary domain, and check the DR metrics to confirm that replication is healthy.

3. Upgrade the primary domain to the same version.

Upgrade the primary domain as soon as you validate the secondary domain. Don't run the domains on different versions longer than necessary.

Clone-based upgrades

A clone-based upgrade creates a cloned cluster of the domain on the new version. After cutover, OpenSearch Service keeps the original cluster, which the upgrade documentation calls the primary cluster, for the retention period that you set. If you don't set a retention period, OpenSearch Service deletes the original cluster immediately after cutover.

If you use a clone-based upgrade, remove the replication relationship before you upgrade either domain. Set up DR again only after both upgrades finish and the original clusters of both domains are deleted. Until a domain's original cluster is deleted, you can roll the domain back to it, which returns the domain to the earlier version. DR doesn't protect the primary domain until you set up DR again and the initial copy finishes.

To upgrade domains with a clone-based upgrade

1. Remove the replication relationship. Both domains become standalone domains and keep their data. For instructions, see Remove the replication relationship.

2. Upgrade the secondary domain, and validate your workload on it.

3. Upgrade the primary domain to the same version.

4. Wait until the original clusters of both domains are deleted. OpenSearch Service deletes each original cluster when its retention period ends. To delete an original cluster sooner, see Delete the primary cluster early.

5. Delete all indexes except system indexes from the secondary domain, or create a new secondary domain that runs the new version. The secondary domain must be empty when you set up DR.

6. Set up DR again. For instructions, see Set up disaster recovery. DR copies all indexes from the primary domain to the secondary domain. The DisasterRecoveryBootstrappingIndices metric returns to 0 after the copy finishes. Copying a large amount of data can take several hours.

Operational guidance

• Rehearse Region moves regularly with a planned switchover, including how your applications discover and connect to the current primary domain. A failover ends the replication relationship and requires a full copy to protect the new primary domain again, so test your failover procedure on non-production domains.

• Keep the domain configurations and non-replicated resources synchronized with infrastructure as code. Use CompareDomainConfiguration to detect differences between the domains.

• When you upgrade the engine version, upgrade the secondary domain first and the primary domain last. If you use a clone-based upgrade, remove the replication relationship first. For more information, see Upgrade domains that use DR.

• Use DescribeDomainReplication and per-index replication status to verify that the secondary domain is current before a planned switchover.

Pricing

There is no additional charge for managed disaster recovery. You pay for both domains at standard Amazon OpenSearch Service rates, and standard AWS data transfer charges apply to the data that DR replicates across Regions. For more information, see Amazon OpenSearch Service pricing on the AWS website.