Upgrading Amazon OpenSearch Service domains - Amazon OpenSearch Service

Upgrading Amazon OpenSearch Service domains

Note

OpenSearch and Elasticsearch version upgrades differ from service software updates. For information on updating the service software for your OpenSearch Service domain, see Service software updates in Amazon OpenSearch Service.

Amazon OpenSearch Service offers in-place upgrades for domains that run OpenSearch 1.0 or later, or Elasticsearch 5.1 or later. If you use services like Amazon Data Firehose or Amazon CloudWatch Logs to stream data to OpenSearch Service, check that these services support the newer version of OpenSearch before migrating.

Supported upgrade paths

Currently, OpenSearch Service supports the following upgrade paths:

From version To version
OpenSearch 1.3 or 2.x

OpenSearch 2.x

Version 2.3 has the following breaking changes:

  • The type parameter was removed from all OpenSearch API endpoints in version 2.0. For more information, see the breaking changes.

  • If your domain contains any indexes (hot, UltraWarm, or cold) that were originally created in Elasticsearch 6.8, those indexes are not compatible with OpenSearch 2.3.

    Before you upgrade to version 2.3, you must reindex the incompatible indexes. For incompatible UltraWarm or cold indexes, migrate them to hot storage, reindex the data, and then migrate them back to warm or cold storage. Alternately, you can delete the indexes if you no longer need them.

    If you accidentally upgrade your domain to version 2.3 without performing these steps first, you won't be able to migrate the incompatible indexes out of their current storage tier. Your only option is to delete them.

OpenSearch 1.x OpenSearch 1.x
Elasticsearch 7.x

Elasticsearch 7.x or OpenSearch 1.x

Important

OpenSearch 1.x introduces numerous breaking changes. For details, see Amazon OpenSearch Service rename - Summary of changes.

Elasticsearch 6.8

Elasticsearch 7.x or OpenSearch 1.x

Important

Elasticsearch 7.0 and OpenSearch 1.0 include numerous breaking changes. Before initiating an in-place upgrade, we recommend taking a manual snapshot of the 6.x domain, restoring it on a test 7.x or OpenSearch 1.x domain, and using that test domain to identify potential upgrade issues. For breaking changes in OpenSearch 1.0, see Amazon OpenSearch Service rename - Summary of changes.

Like Elasticsearch 6.x, indexes can only contain one mapping type, but that type must now be named _doc. As a result, certain APIs no longer require a mapping type in the request body (such as the _bulk API).

For new indexes, self-hosted Elasticsearch 7.x and OpenSearch 1.x have a default shard count of one. OpenSearch Service domains on Elasticsearch 7.x and later retain the previous default of five.

Elasticsearch 6.x

Elasticsearch 6.x

Elasticsearch 5.6

Elasticsearch 6.x

Important

Indexes created in version 6.x no longer support multiple mapping types. Indexes created in version 5.x still support multiple mapping types when restored into a 6.x cluster. Check that your client code creates only a single mapping type per index.

To minimize downtime during the upgrade from Elasticsearch 5.6 to 6.x, OpenSearch Service reindexes the .kibana index to .kibana-6, deletes .kibana, creates an alias named .kibana, and maps the new index to the new alias.

Elasticsearch 5.x Elasticsearch 5.x

The upgrade process consists of three steps:

  1. Pre-upgrade checks – OpenSearch Service checks for issues that can block an upgrade and doesn't proceed to the next step unless these checks succeed.

  2. Snapshot – OpenSearch Service takes a snapshot of the OpenSearch or Elasticsearch cluster and doesn't proceed to the next step unless the snapshot succeeds. If the upgrade fails, OpenSearch Service uses this snapshot to restore the cluster to its original state. For more information see Can't downgrade after upgrade.

  3. Upgrade – OpenSearch Service starts the upgrade, which can take from 15 minutes to several hours to complete. OpenSearch Dashboards might be unavailable during some or all of the upgrade.

Starting an upgrade (console)

The upgrade process is irreversible and can't be paused or cancelled. During an upgrade, you can't make configuration changes to the domain. Before starting an upgrade, double-check that you want to proceed. You can use these same steps to perform the pre-upgrade check without actually starting an upgrade.

If the cluster has dedicated master nodes, OpenSearch upgrades complete without downtime. Otherwise, the cluster might be unresponsive for several seconds post-upgrade while it elects a master node.

To upgrade a domain to a later version of OpenSearch or Elasticsearch
  1. Take a manual snapshot of your domain. This snapshot serves as a backup that you can restore on a new domain if you want to return to using the prior OpenSearch version.

  2. Go to https://aws.amazon.com and choose Sign In to the Console.

  3. Under Analytics, choose Amazon OpenSearch Service.

  4. In the navigation pane, under Domains, choose the domain that you want to upgrade.

  5. Choose Actions and Upgrade.

  6. Select the version to upgrade to. If you're upgrading to an OpenSearch version, the Enable compatibility mode option appears. If you enable this setting, OpenSearch reports its version as 7.10 to allow Elasticsearch OSS clients and plugins like Logstash to continue working with Amazon OpenSearch Service. You can disable this setting later

  7. Choose Upgrade.

  8. Check the Status on the domain dashboard to monitor the status of the upgrade.

Starting an upgrade (CLI)

You can use the following operations to identify the correct version of OpenSearch or Elasticsearch for your domain, start an in-place upgrade, perform the pre-upgrade check, and view progress:

  • get-compatible-versions (GetCompatibleVersions)

  • upgrade-domain (UpgradeDomain)

  • get-upgrade-status (GetUpgradeStatus)

  • get-upgrade-history (GetUpgradeHistory)

For more information, see the AWS CLI command reference and Amazon OpenSearch Service API Reference.

Starting an upgrade (SDK)

This sample uses the OpenSearchService low-level Python client from the AWS SDK for Python (Boto) to check if a domain is eligible for upgrade to a specific version, upgrades it, and continuously checks the upgrade status.

import boto3 from botocore.config import Config import time # Build the client using the default credential configuration. # You can use the CLI and run 'aws configure' to set access key, secret # key, and default Region. DOMAIN_NAME = '' # The name of the domain to upgrade TARGET_VERSION = '' # The version you want to upgrade the domain to. For example, OpenSearch_1.1 my_config = Config( # Optionally lets you specify a Region other than your default. region_name='us-east-1' ) client = boto3.client('opensearch', config=my_config) def check_versions(): """Determine whether domain is eligible for upgrade""" response = client.get_compatible_versions( DomainName=DOMAIN_NAME ) compatible_versions = response['CompatibleVersions'] for i in range(len(compatible_versions)): if TARGET_VERSION in compatible_versions[i]["TargetVersions"]: print('Domain is eligible for upgrade to ' + TARGET_VERSION) upgrade_domain() print(response) else: print('Domain not eligible for upgrade to ' + TARGET_VERSION) def upgrade_domain(): """Upgrades the domain""" response = client.upgrade_domain( DomainName=DOMAIN_NAME, TargetVersion=TARGET_VERSION ) print('Upgrading domain to ' + TARGET_VERSION + '...' + response) time.sleep(5) wait_for_upgrade() def wait_for_upgrade(): """Get the status of the upgrade""" response = client.get_upgrade_status( DomainName=DOMAIN_NAME ) if (response['UpgradeStep']) == 'UPGRADE' and (response['StepStatus']) == 'SUCCEEDED': print('Domain successfully upgraded to ' + TARGET_VERSION) elif (response['StepStatus']) == 'FAILED': print('Upgrade failed. Please try again.') elif (response['StepStatus']) == 'SUCCEEDED_WITH_ISSUES': print('Upgrade succeeded with issues') elif (response['StepStatus']) == 'IN_PROGRESS': time.sleep(30) wait_for_upgrade() def main(): check_versions() if __name__ == "__main__": main()

Troubleshooting validation failures

When you initiate an OpenSearch or Elasticsearch version upgrade, OpenSearch Service first performs a series of validation checks to ensure that your domain is eligible for an upgrade. If any of these checks fail, you receive a notification containing the specific issues that you must fix before upgrading your domain. For a list of potential issues and steps to resolve them, see Troubleshooting validation errors.

Troubleshooting an upgrade

In-place upgrades require healthy domains. Your domain might be ineligible for an upgrade or fail to upgrade for a wide variety of reasons. The following table shows the most common issues.

Issue Description
Optional plugin not supported

When you upgrade a domain with optional plugins, OpenSearch Service automatically upgrades the plugins as well. Therefore, the target version for your domain must also support these optional plugins. If the domain has an optional plugin installed that is not available for the target version, the upgrade request fails.

Too many shards per node OpenSearch, as well as 7.x versions of Elasticsearch, have a default setting of no more than 1,000 shards per node. If a node in your current cluster exceeds this setting, OpenSearch Service doesn't allow you to upgrade. See Exceeded maximum shard limit for troubleshooting options.
Domain in processing The domain is in the middle of a configuration change. Check upgrade eligibility after the operation completes.
Red cluster status One or more indexes in the cluster is red. For troubleshooting steps, see Red cluster status.
High error rate The cluster is returning a large number of 5xx errors when attempting to process requests. This problem is usually the result of too many simultaneous read or write requests. Consider reducing traffic to the cluster or scaling your domain.
Split brain Split brain means that your cluster has more than one master node and has split into two clusters that never will rejoin on their own. You can avoid split brain by using the recommended number of dedicated master nodes. For help recovering from split brain, contact AWS Support.
Master node not found OpenSearch Service can't find the cluster's master node. If your domain uses multi-AZ, an Availability Zone failure might have caused the cluster to lose quorum and be unable to elect a new master node. If the issue does not self-resolve, contact AWS Support.
Too many pending tasks The master node is under heavy load and has many pending tasks. Consider reducing traffic to the cluster or scaling your domain.
Impaired storage volume The disk volume of one or more nodes isn't functioning properly. This issue often occurs alongside other issues, like a high error rate or too many pending tasks. If it occurs in isolation and doesn't self-resolve, contact AWS Support.
KMS key issue The KMS key that is used to encrypt the domain is either inaccessible or missing. For more information, see Monitoring domains that encrypt data at rest.
Snapshot in progress The domain is currently taking a snapshot. Check upgrade eligibility after the snapshot finishes. Also check that you can list manual snapshot repositories, list snapshots within those repositories, and take manual snapshots. If OpenSearch Service is unable to check whether a snapshot is in progress, upgrades can fail.
Snapshot timeout or failure The pre-upgrade snapshot took too long to complete or failed. Check cluster health, and try again. If the problem persists, contact AWS Support.
Incompatible indexes One or more indexes is incompatible with the target version. This problem can occur if you migrated the indexes from an older version of OpenSearch or Elasticsearch. Reindex the indexes and try again.
High disk usage Disk usage for the cluster is above 90%. Delete data or scale the domain, and try again.
High JVM usage JVM memory pressure is above 75%. Reduce traffic to the cluster or scale the domain, and try again.
OpenSearch Dashboards alias problem .dashboards is already configured as an alias and maps to an incompatible index, likely one from an earlier version of OpenSearch Dashboards. Reindex and try again.
Red Dashboards status OpenSearch Dashboards status is red. Try using Dashboards when the upgrade completes. If the red status persists, resolve it manually, and try again.
Cross-cluster compatibility

You can only upgrade if cross-cluster compatibility is maintained between the source and destination domains after the upgrade. During the upgrade process, any incompatible connections are identified. To proceed, either upgrade the remote domain or delete the incompatible connections. Note that if replication is active on the domain, you can't resume it once you delete the connection.

Other OpenSearch Service service issue Issues with OpenSearch Service itself might cause your domain to display as ineligible for an upgrade. If none of the preceding conditions apply to your domain and the problem persists for more than a day, contact AWS Support.

Using a snapshot to migrate data

In-place upgrades are the easier, faster, and more reliable way to upgrade a domain to a later OpenSearch or Elasticsearch version. Snapshots are a good option if you need to migrate from a pre-5.1 version of Elasticsearch or want to migrate to an entirely new cluster.

The following table shows how to use snapshots to migrate data to a domain that uses a different OpenSearch or Elasticsearch version. For more information about taking and restoring snapshots, see Creating index snapshots in Amazon OpenSearch Service.

From version To version Migration process
OpenSearch 1.3 or 2.x OpenSearch 2.x
  1. Review breaking changes for OpenSearch 2.3 to see if you need to make adjustments to your indexes or applications.

  2. Create a manual snapshot of the 1.3 or 2.x domain.

  3. Create a 2.x domain that's a higher version than your original 1.3 or 2.x domain.

  4. Restore the snapshot from the original domain to the 2.x domain. During the operation, you might need to restore your .opensearch index under a new name:

    POST _snapshot/<repository-name>/<snapshot-name>/_restore { "indices": "*", "ignore_unavailable": true, "rename_pattern": ".opensearch", "rename_replacement": ".backup-opensearch" }

    Then you can reindex .backup-opensearch on the new domain and alias it to .opensearch. Note that the _restore REST call doesn't include include_global_state because the default in _restore is false. As a result, the test domain won't include any index templates and won't have the full state from the backup.

  5. If you no longer need your original domain, delete it. Otherwise, you continue to incur charges for the domain.

OpenSearch 1.x OpenSearch 1.x
  1. Create a manual snapshot of the 1.x domain.

  2. Create a 1.x domain that's a higher version than your original 1.x domain.

  3. Restore the snapshot from the original domain to the new 1.x domain. During the operation, you might need to restore your .opensearch index under a new name:

    POST _snapshot/<repository-name>/<snapshot-name>/_restore { "indices": "*", "ignore_unavailable": true, "rename_pattern": ".opensearch", "rename_replacement": ".backup-opensearch" }

    Then you can reindex .backup-opensearch on the new domain and alias it to .opensearch. Note that the _restore REST call doesn't include include_global_state because the default in _restore is false. As a result, the test domain won't include any index templates and won't have the full state from the backup.

  4. If you no longer need your original domain, delete it. Otherwise, you continue to incur charges for the domain.

Elasticsearch 6.x or 7.x OpenSearch 1.x
  1. Review breaking changes for OpenSearch 1.0 to see if you need to make adjustments to your indexes or applications.

  2. Create a manual snapshot of the Elasticsearch 7.x or 6.x domain.

  3. Create an OpenSearch 1.x domain.

  4. Restore the snapshot from the Elasticsearch domain to the OpenSearch domain. During the operation, you might need to restore your .elasticsearch index under a new name:

    POST _snapshot/<repository-name>/<snapshot-name>/_restore { "indices": "*", "ignore_unavailable": true, "rename_pattern": ".elasticsearch", "rename_replacement": ".backup-opensearch" }

    Then you can reindex .backup-opensearch on the new domain and alias it to .elasticsearch. Note that the _restore REST call doesn't include include_global_state because the default in _restore is false. As a result, the test domain won't include any index templates and won't have the full state from the backup.

  5. If you no longer need your original domain, delete it. Otherwise, you continue to incur charges for the domain.

Elasticsearch 6.x Elasticsearch 7.x
  1. Review breaking changes for 7.0 to see if you need to make adjustments to your indexes or applications.

  2. Create a manual snapshot of the 6.x domain.

  3. Create a 7.x domain.

  4. Restore the snapshot from the original domain to the 7.x domain. During the operation, you likely need to restore the .opensearch index under a new name:

    POST _snapshot/<repository-name>/<snapshot-name>/_restore { "indices": "*", "ignore_unavailable": true, "rename_pattern": ".elasticsearch", "rename_replacement": ".backup-elasticsearch" }

    Then you can reindex .backup-elasticsearch on the new domain and alias it to .elasticsearch. Note that the _restore REST call doesn't include include_global_state because the default in _restore is false. As a result, the test domain won't include any index templates and won't have the full state from the backup.

  5. If you no longer need your original domain, delete it. Otherwise, you continue to incur charges for the domain.

Elasticsearch 6.x Elasticsearch 6.8
  1. Create a manual snapshot of the 6.x domain.

  2. Create a 6.8 domain.

  3. Restore the snapshot from the original domain to the 6.8 domain.

  4. If you no longer need your original domain, delete it. Otherwise, you continue to incur charges for the domain.

Elasticsearch 5.x Elasticsearch 6.x
  1. Review breaking changes for 6.0 to see if you need to make adjustments to your indices or applications.

  2. Create a manual snapshot of the 5.x domain.

  3. Create a 6.x domain.

  4. Restore the snapshot from the original domain to the 6.x domain.

  5. If you no longer need your 5.x domain, delete it. Otherwise, you continue to incur charges for the domain.

Elasticsearch 5.x Elasticsearch 5.6
  1. Create a manual snapshot of the 5.x domain.

  2. Create a 5.6 domain.

  3. Restore the snapshot from the original domain to the 5.6 domain.

  4. If you no longer need your original domain, delete it. Otherwise, you continue to incur charges for the domain.

Elasticsearch 2.3 Elasticsearch 6.x

Elasticsearch 2.3 snapshots are not compatible with 6.x. To migrate your data directly from 2.3 to 6.x, you must manually recreate your indexes in the new domain.

Alternately, you can follow the 2.3 to 5.x steps in this table, perform _reindex operations in the new 5.x domain to convert your 2.3 indexes to 5.x indexes, and then follow the 5.x to 6.x steps.

Elasticsearch 2.3 Elasticsearch 5.x
  1. Review breaking changes for 5.0 to see if you need to make adjustments to your indexes or applications.

  2. Create a manual snapshot of the 2.3 domain.

  3. Create a 5.x domain.

  4. Restore the snapshot from the 2.3 domain to the 5.x domain.

  5. If you no longer need your 2.3 domain, delete it. Otherwise, you continue to incur charges for the domain.

Elasticsearch 1.5 Elasticsearch 5.x

Elasticsearch 1.5 snapshots are not compatible with 5.x. To migrate your data from 1.5 to 5.x, you must manually recreate your indexes in the new domain.

Important

1.5 snapshots are compatible with 2.3, but OpenSearch Service 2.3 domains do not support the _reindex operation. Because you cannot reindex them, indexes that originated in a 1.5 domain still fail to restore from 2.3 snapshots to 5.x domains.

Elasticsearch 1.5 Elasticsearch 2.3
  1. Use the migration plugin to find out if you can directly upgrade to version 2.3. You might need to make changes to your data before migration.

    1. In a web browser, open http://domain-endpoint/_plugin/migration/.

    2. Choose Run checks now.

    3. Review the results and, if needed, follow the instructions to make changes to your data.

  2. Create a manual snapshot of the 1.5 domain.

  3. Create a 2.3 domain.

  4. Restore the snapshot from the 1.5 domain to the 2.3 domain.

  5. If you no longer need your 1.5 domain, delete it. Otherwise, you continue to incur charges for the domain.