Menu
Amazon Elasticsearch Service
Developer Guide (API Version 2015-01-01)

Creating and Configuring Amazon Elasticsearch Service Domains

This chapter describes how to create and configure Amazon Elasticsearch Service (Amazon ES) domains. An Amazon ES domain is the hardware, software, and data exposed by Amazon Elasticsearch Service endpoints.

Unlike the brief instructions in the Getting Started tutorial, this chapter describes all options and provides relevant reference information. You can complete each procedure by using instructions for the Amazon ES console, the AWS Command Line Interface (AWS CLI), or the AWS SDKs.

Creating Amazon ES Domains

This section describes how to create Amazon ES domains by using the Amazon ES console or by using the AWS CLI with the create-elasticsearch-domain command. The procedures for the AWS CLI include syntax and examples.

Creating Amazon ES Domains (Console)

Use the following procedure to create an Amazon ES domain by using the console.

To create an Amazon ES domain (console)

  1. Go to https://aws.amazon.com, and then choose Sign In to the Console.

  2. Under Analytics, choose Elasticsearch Service.

  3. Choose Create a new domain.

    Alternatively, choose Get Started if this is the first Amazon ES domain that you will create for your AWS account.

  4. On the Define domain page, for Domain name, type a name for your domain.

  5. For Version, choose an Elasticsearch version for your domain. We recommend that you choose version 5.1. For more information, see Choosing an Elasticsearch Version.

  6. Choose Next.

  7. For Instance count, choose the number of instances that you want.

    The default is one. The default limit is 20 instances per domain. To request an increase up to 100 instances per domain, create a case with the AWS Support Center. We recommend that you choose more than two instances to avoid potential Elasticsearch issues, such as the split brain issue. We also recommend that you have a replica for each index to avoid potential data loss. For more information about replicas, see Shards and Replicas in the Elasticsearch documentation.

    Note

    You can use a T2 instance type only if the instance count for the domain is 10 or fewer.

  8. For Instance type, choose an instance type for the data nodes.

    To see a list of the instance types that Amazon ES supports, see Supported Instance Types.

    Note

    • The t2.micro.elasticsearch instance is supported only with Elasticsearch version 2.3 or 1.5.

    • The M3 instance type is not available in the us-east-2, ca-central-1, eu-west-2, ap-northeast-2, and ap-south-1 regions.

    • The I2 instance type is not available in the sa-east-1, ca-central-1, eu-west-2, and us-east-2 regions.

    • The R3 instance type is not available in the ca-central-1, eu-west-2, and sa-east-1 regions.

  9. If you need to ensure cluster stability or if you have a domain that has more than 10 instances, enable a dedicated master node. Dedicated master nodes increase cluster stability and are required for a domain that has an instance count greater than 10. For more information, see About Dedicated Master Nodes.

    1. Select the Enable dedicated master check box.

    2. For Dedicated master instance type, choose an instance type for the dedicated master node.

      For a list of the instance types that Amazon ES supports, see Supported Instance Types.

      Note

      • You can choose an instance type for the dedicated master node that differs from the instance type that you choose for the data nodes.

      • You can use a T2 instance type only if the instance count is 10 or fewer.

    3. For Dedicated master instance count, choose the number of instances for the dedicated master node.

      We recommend choosing an odd number of instances to avoid potential Elasticsearch issues, such as the split brain issue. The default and recommended number is three.

  10. (Optional) To provide high availability for data nodes, select the Enable zone awareness check box.

    Zone awareness distributes Amazon ES data nodes across two Availability Zones in the same region. If you enable zone awareness, you must have an even number of instances in the instance count, and you also must use the Amazon ES API to replicate your data for your cluster. This allows for the even distribution of shards across two Availability Zones. For more information, see Enabling Zone Awareness.

  11. For Storage type, choose either Instance (the default) or EBS.

    Use an EBS volume for storage rather than the storage attached to the selected instance type if your Amazon ES domain requires more storage. Domains with very large indices or large numbers of indices often benefit from the increased storage capacity of EBS volumes. If you choose EBS, the following boxes appear.

    1. For EBS volume type, choose an EBS volume type.

      If you choose Provisioned IOPS (SSD) for the EBS volume type, for Provisioned IOPS, type the baseline IOPS performance that you want. For more information, see Amazon EBS Volumes in the Amazon EC2 documentation.

    2. For EBS volume size, type the size of the EBS volume that you want to attach to each data node.

      Calculate the total amount of EBS-based storage for the Amazon ES domain using the following formula: (number of data nodes) * (EBS volume size). The size of an EBS volume depends on both the specified EBS volume type and the instance type to which it is attached. For information about the minimum and maximum size of supported EBS volumes in an Amazon ES domain, see EBS Volume Size Limits. For more information about supported EBS volume types and sizes, see Configuring EBS-based Storage.

  12. For Automated snapshot start hour, choose the hour for automated daily snapshots of domain indices.

    By default, the service takes an automated snapshot within an hour of midnight.

  13. (Optional) Choose Advanced options.

    1. (Optional) If you want to configure access to domain sub-resources, for rest.action.multi.allow_explicit_index, choose false.

      Disabling this property prevents users from bypassing access control for sub-resources. For more information about access control, see URL-based access control in the Elasticsearch documentation. For more information about access policies for sub-resources, see Configuring Access Policies.

    2. (Optional) For indices.fielddata.cache.size, specify the percentage of heap space to allocate to the field data cache.

      By default, this setting is unbounded. For more information about the field data cache, see Field data in the Elasticsearch documentation.

      Note

      Many customers query rotating daily indices. We recommend that you begin benchmark testing with indices.fielddata.cache.size configured to 40% of the JVM heap for most of these use cases. However, if you have very large indices, you might need a larger field data cache.

  14. Choose Next.

  15. On the Set up access policy page, select a pre-configured policy from the Select a template dropdown list and edit it to meet the needs of your domain. Alternatively, you can add one or more Identity and Access Management (IAM) policy statements in the Edit the access policy box.

    Amazon Elasticsearch Service offers several ways to configure access to your Amazon ES domains. The console provides preconfigured access policies that you can customize to the specific needs of your domain, as well as the ability to import access policies from other Amazon ES domains. The service also allows you to specify separate, fine-grained access policies to each domain sub-resource. For example, you can assign a different policy to each index in your Amazon ES domain. For more information, see Configuring Access Policies.

  16. Choose Next.

  17. On the Review page, review your domain configuration, and then choose Confirm and create.

  18. Choose OK.

Note

New domains take up to ten minutes to initialize. After your domain is initialized, you can upload data and make changes to the domain.

Creating Amazon ES Domains (AWS CLI)

Instead of creating an Amazon ES domain by using the console, you can create a domain by using the AWS CLI. Use the following syntax to create an Amazon ES domain.

Syntax

aws es create-elasticsearch-domain --domain-name <value>

[--elasticsearch-version <value>]

[--elasticsearch-cluster-config <value>]

[--ebs-options <value>]

[--access-policies <value>]

[--snapshot-options <value>]

[--advanced-options <value>]

[--cli-input-json <value>]

[--generate-cli-skeleton]

The following table provides more information about each of the optional parameters.

Optional Parameters

Parameter Description
--elasticsearch-version Specifies the Elasticsearch version of the domain. Currently, Amazon ES supports versions 1.5, 2.3, and 5.1. If not specified, the default value is 1.5. For more information, see Choosing an Elasticsearch Version.
--elasticsearch-cluster-config Specifies the instance type and count of the domain, whether zone awareness is enabled, and whether the domain uses a dedicated master node. Dedicated master nodes increase cluster stability and are required for a domain that has an instance count greater than 10. For more information, see Configuring Amazon ES Domains.
--ebs-options Specifies whether the domain uses an EBS volume for storage. If true, this parameter must also specify the EBS volume type, size, and, if applicable, IOPS value. For more information, see Configuring EBS-based Storage.
--access-policies Specifies the access policy for the domain. For more information, see Configuring Access Policies.
--snapshot-options Specifies the hour in UTC during which the service performs a daily automated snapshot of the indices in the domain. The default value is 0, or midnight, which means that the snapshot is taken anytime between midnight and 1:00 AM. For more information, see Configuring Snapshots.
--advanced-options Specifies whether to allow references to indices in the bodies of HTTP request objects. For more information, see Configuring Advanced Options.
--generate-cli-skeleton Displays JSON for all specified parameters. Save the output to a file so that you can later read the file with the --cli-input-json parameter rather than typing the parameters at the command line. For more information, see Generate CLI Skeleton and CLI Input JSON Parameters in the AWS Command Line Interface User Guide.
--cli-input-json Specifies the name of a JSON file that contains a set of CLI parameters. For more information, see Generate CLI Skeleton and CLI Input JSON Parameters in the AWS Command Line Interface User Guide.

Examples

The first example demonstrates the following Amazon ES domain configuration:

  • Creates an Amazon ES domain named weblogs with Elasticsearch version 5.1

  • Populates the domain with two instances of the m4.large.elasticsearch instance type

  • Uses a 100 GB Magnetic disk EBS volume for storage for each data node

  • Allows anonymous access, but only from a single IP address: 192.0.2.0/32

Copy
aws es create-elasticsearch-domain --domain-name weblogs --elasticsearch-version 5.1 --elasticsearch-cluster-config InstanceType=m4.large.elasticsearch,InstanceCount=2 --ebs-options EBSEnabled=true,VolumeType=standard,VolumeSize=100 --access-policies '{"Version": "2012-10-17", "Statement": [{"Action": "es:*", "Principal":"*","Effect": "Allow", "Condition": {"IpAddress":{"aws:SourceIp":["192.0.2.0/32"]}}}]}'

The next example demonstrates the following Amazon ES domain configuration:

  • Creates an Amazon ES domain named weblogs with Elasticsearch version 5.1

  • Populates the domain with six instances of the m4.large.elasticsearch instance type

  • Uses a 100 GB General Purpose (SSD) EBS volume for storage for each data node

  • Restricts access to the service to a single user, identified by the user's AWS account ID: 555555555555

  • Enables zone awareness

Copy
aws es create-elasticsearch-domain --domain-name weblogs --elasticsearch-version 5.1 --elasticsearch-cluster-config InstanceType=m4.large.elasticsearch,InstanceCount=6,ZoneAwarenessEnabled=true --ebs-options EBSEnabled=true,VolumeType=gp2,VolumeSize=100 --access-policies '{"Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": {"AWS": "arn:aws:iam::555555555555:root" }, "Action":"es:*", "Resource": "arn:aws:es:us-east-1:555555555555:domain/logs/*" } ] }'

The next example demonstrates the following Amazon ES domain configuration:

  • Creates an Amazon ES domain named weblogs with Elasticsearch version 5.1

  • Populates the domain with ten instances of the m4.xlarge.elasticsearch instance type

  • Populates the domain with three instances of the m4.large.elasticsearch instance type to serve as dedicated master nodes

  • Uses a 100 GB Provisioned IOPS EBS volume for storage, configured with a baseline performance of 1000 IOPS for each data node

  • Restricts access to a single user and to a single sub-resource, the _search API

  • Configures automated daily snapshots of the indices for 03:00 UTC

Copy
aws es create-elasticsearch-domain --domain-name weblogs --elasticsearch-version 5.1 --elasticsearch-cluster-config InstanceType=m4.xlarge.elasticsearch,InstanceCount=10,DedicatedMasterEnabled=true,DedicatedMasterType=m4.large.elasticsearch,DedicatedMasterCount=3 --ebs-options EBSEnabled=true,VolumeType=io1,VolumeSize=100,Iops=1000 --access-policies '{"Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::555555555555:root" }, "Action": "es:*", "Resource": "arn:aws:es:us-east-1:555555555555:domain/mylogs/_search" } ] }' --snapshot-options AutomatedSnapshotStartHour=3

Note

If you successfully create an Amazon ES domain, the CLI ignores attempts to create another domain with same name. The CLI does not report an error.

Creating Amazon ES Domains (AWS SDKs)

The AWS SDKs (except the Android and iOS SDKs) support all the actions defined in the Amazon ES Configuration API Reference, including create-elasticsearch-domain. For more information about installing and using the AWS SDKs, see AWS Software Development Kits.

Configuring Amazon ES Domains

Update your Amazon ES domain configuration with any of the following changes to meet the demands of increased traffic and data:

  • Change the instance count

  • Change the instance type

  • Enable or disable a dedicated master node

  • Enable or disable zone awareness

  • Configure EBS-based storage

  • Change the start time for automated snapshots of domain indices

  • Configure a native Amazon ES property

Note

For information about configuring a domain to use an EBS volume for storage, see Configuring EBS-based Storage.

Configuring Amazon ES Domains (Console)

Use the following procedure to make updates to your Amazon ES configuration by using the console.

To configure an Amazon ES domain (console)

  1. Go to https://aws.amazon.com, and then choose Sign In to the Console.

  2. Under Analytics, choose Elasticsearch Service.

  3. In the navigation pane, under My domains, choose the domain that you want to update.

  4. Choose Configure cluster.

  5. On the Configure cluster page, update the configuration of the domain.

    The cluster is a collection of one or more data nodes, optional dedicated master nodes, and storage required to run Amazon ES and operate your domain.

    1. If you want to change the instance type for data nodes, for Instance type, choose a new instance type.

      To see a list of the instance types that Amazon ES supports, see Supported Instance Types.

      Note

      • The t2.micro.elasticsearch instance is supported only with Elasticsearch version 2.3 or 1.5.

      • The M3 instance type is not available in the us-east-2, ca-central-1, eu-west-2, ap-northeast-2, and ap-south-1 regions.

      • The I2 instance type is not available in the sa-east-1, ca-central-1, eu-west-2, and us-east-2 regions.

      • The R3 instance type is not available in the ca-central-1, eu-west-2, and sa-east-1 regions.

    2. If you want to change the instance count, for Instance count, choose an integer from one to twenty. To request an increase up to 100 instances per domain, create a case with the AWS Support Center.

    3. If you want to improve cluster stability or if your domain has an instance count greater than 10, enable a dedicated master node for your cluster. For more information, see About Dedicated Master Nodes.

      1. Select the Enable dedicated master check box.

      2. For Dedicated master instance type, choose an instance type for the dedicated master node.

        You can choose an instance type for the dedicated master node that differs from the instance type that you choose for the data nodes.

        To see a list of the instance types that Amazon ES supports, see Supported Instance Types.

      3. For Dedicated master instance count, choose the number of instances for the dedicated master node.

        We recommend choosing an odd number of instances to avoid potential Amazon ES issues, such as the split brain issue. The default and recommended number is three.

    4. If you want to enable zone awareness, select the Enable zone awareness check box. If you enable zone awareness, you must have an even number of instances in your instance count. This allows for the even distribution of shards across two Availability Zones in the same region.

    5. If you want to change the hour during which the service takes automated daily snapshots of the primary index shards of your Amazon ES domain, for Automated snapshot start hour, choose an integer.

    6. Choose Advanced options.

      1. (Optional) If you want to configure access to domain sub-resources, for rest.action.multi.allow_explicit_index, choose false.

        Disabling this property prevents users from bypassing access control for sub-resources. For more information about access control, see URL-based access control in the Elasticsearch documentation. For more information about access policies for sub-resources, see Configuring Access Policies.

      2. For rest.action.multi.allow_explicit_index, choose false.

      3. (Optional) For indices.fielddata.cache.size, specify the percentage of heap space to allocate to the field data cache.

        By default, this setting is unbounded. For more information about the field data cache, see Field data in the Elasticsearch documentation.

        Note

        Many customers query rotating daily indices. We recommend that you begin benchmark testing with indices.fielddata.cache.size configured to 40% of the JVM heap for most of these cases. However, if you have very large indices you might need a larger field data cache.

    7. Choose Submit.

Configuring Amazon ES Domains (AWS CLI)

Use the elasticsearch-cluster-config option to configure your Amazon ES cluster by using the AWS CLI. The following syntax is used by both the create-elasticsearch-domain and update-elasticsearch-domain-config commands.

Syntax

Copy
--elasticsearch-cluster-config InstanceType=<value>,InstanceCount=<value>,DedicatedMasterEnabled=<value>,DedicatedMasterType=<value>,DedicatedMasterCount=<value>,ZoneAwarenessEnabled=<value>

Note

Do not include spaces between parameters for the same option.

The following table describes the parameters in more detail.

Parameter Valid Values Description
InstanceType Any supported instance type. See Supported Instance Types. The hardware configuration of the computer that will host the instance. The default is m4.large.elasticsearch.
InstanceCount Integer The number of instances in the Amazon ES domain. The default is one, and the maximum default limit is twenty. To request an increase up to 100 instances per domain, create a case with the AWS Support Center.
DedicatedMasterEnabled true or false Specifies whether to use a dedicated master node for the Amazon ES domain. The default value is false.
DedicatedMasterType Any supported instance type The hardware configuration of the computer that will host the master node. The default is m4.large.elasticsearch.
DedicatedMasterCount Integer The number of instances used for the dedicated master node. The default is three.
ZoneAwarenessEnabled true or false Specifies whether to enable zone awareness for the Amazon ES domain. The default value is false.

Note

  • The t2.micro.elasticsearch instance is supported only with Elasticsearch version 2.3 or 1.5.

  • The M3 instance type is not available in the us-east-2, ca-central-1, eu-west-2, ap-northeast-2, and ap-south-1 regions.

  • The I2 instance type is not available in the sa-east-1, ca-central-1, eu-west-2, and us-east-2 regions.

  • The R3 instance type is not available in the ca-central-1, eu-west-2, and sa-east-1 regions.

Examples

The following example creates an Amazon ES domain named mylogs with Elasticsearch version 5.1 with two instances of the m4.large.elasticsearch instance type and zone awareness enabled:

Copy
aws es create-elasticsearch-domain --domain-name mylogs --elasticsearch-version 5.1 --elasticsearch-cluster-config InstanceType=m4.large.elasticsearch,InstanceCount=2,DedicatedMasterEnabled=false,ZoneAwarenessEnabled=true

However, you likely will want to reconfigure your new Amazon ES domain as network traffic grows and as the quantity and size of documents increase. For example, you might decide to use a larger instance type, use more instances, and enable a dedicated master node. The following example updates the domain configuration with these changes:

Copy
aws es update-elasticsearch-domain-config --domain-name mylogs --elasticsearch-cluster-config InstanceType=m4.xlarge.elasticsearch,InstanceCount=3,DedicatedMasterEnabled=true,DedicatedMasterType=m4.large.elasticsearch,DedicatedMasterCount=3

Configuring Amazon ES Domains (AWS SDKs)

The AWS SDKs (except the Android and iOS SDKs) support all the actions defined in the Amazon ES Configuration API Reference, including update-elasticsearch-domain-config. For more information about installing and using the AWS SDKs, see AWS Software Development Kits.

Configuring EBS-based Storage

An Amazon EBS volume is a block-level storage device that you can attach to a single instance. EBS volumes enable you to independently scale the storage resources of your Amazon ES domain from its compute resources. EBS volumes are most useful for domains with very large data sets, but without the need for large compute resources. EBS volumes are much larger than the default storage provided by the instance. Amazon Elasticsearch Service supports the following EBS volume types:

  • General Purpose (SSD)

  • Provisioned IOPS (SSD)

  • Magnetic

Note

When changing an EBS volume type from provisioned IOPS to non-provisioned EBS volume types, set the IOPS value to 0.

Caution

Currently, if the data node that is attached to an EBS volume fails, the EBS volume also fails.

Configuring EBS-based Storage (Console)

Use the following procedure to enable EBS-based storage by using the console.

To enable EBS-based storage (console)

  1. Go to https://aws.amazon.com, and then choose Sign In to the Console.

  2. Under Analytics, choose Elasticsearch Service.

  3. In the navigation pane, under My domains, choose the domain that you want to configure.

  4. Choose Configure cluster.

  5. For Storage type, choose EBS.

  6. For EBS volume type, choose an EBS volume type.

    • If you choose Provisioned IOPS (SSD) for the EBS volume type, for Provisioned IOPS, type the baseline IOPS performance that you want.

  7. For EBS volume size, type the size that you want for the EBS volume.

    Calculate the total amount of EBS-based storage for the Amazon ES domain using the following formula: (number of data nodes) * (EBS volume size). The size of an EBS volume depends on both the specified EBS volume type and the instance type to which it is attached. For information about the minimum and maximum size of supported EBS volumes in an Amazon ES domain, see EBS Volume Size Limits. For more information about supported EBS volume types and sizes, see Configuring EBS-based Storage.

  8. Choose Submit.

Note

You must set the IOPS value for a Provisioned IOPS EBS volume to no more than 30 times the maximum storage of the volume. For example, if your volume has a maximum size of 100 GB, you can't assign an IOPS value for it that is greater than 3000.

For more information, see Amazon EBS Volumes in the Amazon EC2 documentation.

Configuring EBS-based Storage (AWS CLI)

Use the --ebs-options option to configure EBS-based storage by using the AWS CLI. The following syntax is used by both the create-elasticsearch-domain and update-elasticsearch-domain-config commands.

Syntax

--ebs-options EBSEnabled=<value>,VolumeType=<value>,VolumeSize=<value>,IOPS=<value>

Parameter Valid Values Description
EBSEnabled true or false Specifies whether to use an EBS volume for storage rather than the storage provided by the instance. The default value is false.
VolumeType Any of the following:
  • gp2 (General Purpose SSD)

  • io1 (Provisioned IOPS SSD)

  • standard (Magnetic)

The EBS volume type to use with the Amazon ES domain.
VolumeSize Integer Specifies the size of the EBS volume for each data node. The minimum and maximum size of an EBS volume depends on both the specified EBS volume type and the instance type to which it is attached. To see a table that shows the minimum and maximum EBS size for each instance type, see Service Limits.
IOPS Integer Specifies the baseline I/O performance for the EBS volume. This parameter is used only by Provisioned IOPS (SSD) volumes. The minimum value is 1000. The maximum value is 16000.

Note

We recommend that you do not set the IOPS value for a Provisioned IOPS EBS volume to more than 30 times the maximum storage of the volume. For example, if your volume has a maximum size of 100 GB, you should not assign an IOPS value for it that is greater than 3000. For more information, including use cases for each volume types, see Amazon EBS Volume Types in the Amazon EC2 documentation.

Examples

The following example creates a domain named mylogs with Elasticsearch version 5.1 with a 10 GB General Purpose EBS volume:

Copy
aws es create-elasticsearch-domain --domain-name=mylogs --elasticsearch-version 5.1 --ebs-options EBSEnabled=true,VolumeType=gp2,VolumeSize=10

However, you might need a larger EBS volume as the size of your search indices increases. For example, you might opt for a 100 GB Provisioned IOPS volume with a baseline I/O performance of 3000 IOPS. The following example updates the domain configuration with those changes:

Copy
aws es update-elasticsearch-domain-config --domain-name=mylogs --ebs-options EBSEnabled=true,VolumeType=io1,VolumeSize=100,IOPS=3000

Configuring EBS-based Storage (AWS SDKs)

The AWS SDKs (except the Android and iOS SDKs) support all the actions defined in the Amazon ES Configuration API Reference, including the --ebs-options parameter to the update-elasticsearch-domain-config command. For more information about installing and using the AWS SDKs, see AWS Software Development Kits.

Configuring Access Policies

Amazon Elasticsearch Service offers several ways to configure access to your Amazon ES domains. The console provides preconfigured access policies that you can customize to the specific needs of your domain as well as the ability to import access policies from other Amazon ES domains. The service also allows you to specify separate, fine-grained access policies to each domain sub-resource. For example, you can assign a different policy to each index in your Amazon ES domain.

Access Configuration Method Description
Resource-based access policy Resource-based access policies are attached to a specific Amazon ES domain. A resource-based policy specifies who can access the endpoint of the domain. Use the Principal policy element to specify who is allowed access. Use the Resource policy element to specify which resources are accessible.
IP-based policy IP-based access policies restrict access to an Amazon ES domain to one or more specific IP addresses. IP-based policies also can be configured to allow anonymous access, which enables you to submit unsigned requests to an Amazon ES domain. Use the Condition policy element to specify which IP addresses are allowed to access the service.
IAM user and role-based access policies Amazon ES also supports access policies based on IAM users and roles. Use the IAM service to specify which users and roles can access the service and what sub-resources they can use.

Configuring Access Policies (Console)

Use the following procedure to configure access policies by using the console.

To configure access policies (console)

  1. Go to https://aws.amazon.com, and then choose Sign In to the Console.

  2. Under Analytics, choose Elasticsearch Service.

  3. In the navigation pane, under My domains, choose the domain that you want to update.

  4. Choose Modify access policy.

  5. Edit the access policy.

    Alternatively, choose one of the policy templates from the Select a template dropdown list, and then edit it as needed for your domain.

    Preconfigured Access Policy Description
    Allow or deny access to one or more AWS accounts or IAM users This policy is used to allow or deny access to one or more AWS accounts or IAM users.
    Allow open access to the domain

    This policy is not recommended because it allows anyone to delete, modify, or access indexes and documents in your domain. It is intended only as a convenience for testing. Don't load sensitive data to a domain that has these settings.

    Deny access to the domain This policy allows access only through the Amazon ES console or by the owner of the AWS account who created the domain.
    Allow access to the domain from specific IP(s) This policy is used to restrict anonymous access to a specific IP address or range of IP addresses.
    Copy access policy from another domain This policy provides a convenient way to import an existing access policy from another domain.

  6. Choose Submit.

Configuring Access Policies (AWS CLI)

Use the --access-policies option to configure access policies by using the AWS CLI. The following syntax is used by both the create-elasticsearch-domain and update-elasticsearch-domain-config commands.

Syntax

Copy
--access-policies=<value>

Parameter Valid Values Description
--access-policies JSON Specifies the access policy for the Amazon ES domain.

Amazon Elasticsearch Service supports all the policy elements that are documented in the IAM Policy Elements Reference. The following table shows the most common elements.

Valid Values JSON Policy Element Description
The current version of the policy language is 2012-10-17. All access policies should specify this value. Version Specifies the language version for the access policy.
Allow or Deny Effect Specifies whether the statement allows or blocks access to the specified actions.
Any string Sid A descriptive name for the policy statement. This field is optional.
Amazon ES supports the following actions for HTTP methods. You can attach a separate access policy to each HTTP method:
  • es:ESHttpDelete

  • es:ESHttpGet

  • es:ESHttpHead

  • es:ESHttpPost

  • es:ESHttpPut

Amazon ES also supports the following actions for the service configuration APIs:

  • es:CreateElasticsearchDomain

  • es:DescribeElasticsearchDomain

  • es:DescribeElasticsearchDomains

  • es:DescribeElasticsearchDomainConfig

  • es:DeleteElasticsearchDomain

  • es:ListDomainNames

  • es:AddTags

  • es:ListTags

  • es:RemoveTags

  • es:UpdateElasticsearchDomainConfig

For a description of each API, including the HTTP request method required for each, see the Amazon ES Configuration API Reference.

Action Specifies the Amazon ES actions to which the access policy applies. Assign a value of "Action":"es:*" to allow full access to the domain endpoint with any HTTP method.
Use the following syntax to specify domain resources for Amazon ES:

arn:aws:es:<region>:<aws_account_id>:domain/<domain-name>/<sub-resource>

Specify the wildcard (*) as a sub-resource to allow or deny access to all sub-resources:

arn:aws:es:<region>:<aws_account_id>:domain/<domain-name>/*

Use the following syntax to allow or deny all access to a specific sub-resource:

arn:aws:es:<region>:<aws_account_id>:domain/<domain-name>/<sub-resource>/*

Amazon ES allows you to define a different access policy for each sub-resource, such as indices. You also can define a different access policy for each Amazon ES API. For example, you can limit the scope of the grant permission to only the Amazon ES _search API.

arn:aws:es:us-east-1:<account-id>:domain/weblogs/_search

Resource Specifies the specific object or objects to which the access policy applies.
Amazon ES supports all the conditions described in Available Global Condition Keys in the Using IAM guide. Condition Specifies conditions that determine when the access policy is in effect. When configuring anonymous, IP-based access, specify the IP addresses for which the access rule applies. For example: "IpAddress": {"aws:SourceIp": ["192.0.2.0/32"]}.
Any of the following:
  • Other AWS accounts

    "Principal":{"AWS":["arn:aws:iam::<aws_account_id>:root"]}

  • IAM users

    "Principal":{"AWS": [arn:aws:iam::<aws_account_id>:user/<username>}

Principal Specifies the AWS account or IAM user that is allowed or denied access to a resource. Specifying a wildcard (*) enables anonymous access to the domain, which is not recommended. If you do enable anonymous access, we strongly recommend that you add an IP-based condition to restrict which IP addresses can submit requests to the Amazon ES domain.

For more information, see Principal in the IAM Policy Elements Reference.

Resource-based Policy Example

The following example of a resource-based access policy restricts access to the service to a single user, identified by the user's AWS account ID, 555555555555, in the Principal policy element. This user is granted access to the index1 domain sub-resource, but will not be able to access other indices in the domain.

Copy
aws es update-elasticsearch-domain-config --domain-name mylogs --access-policies '{"Version": "2012-10-17", "Statement": [ { "Effect": "Allow","Principal": {"AWS": "arn:aws:iam::123456789012:root" },"Action":"es:*","Resource":"arn:aws:es:us-east-1:555555555555:domain/index1/*" } ] }'

IP-based Policy Example

The following example of an IP-based access policy allows anonymous access, but restricts that access to a single range of IP addresses:

Copy
aws es update-elasticsearch-domain-config --domain-name mylogs --access-policies '{"Version": "2012-10-17","Statement": [{"Action":"es:*","Principal":"*","Effect":"Allow","Condition": {"IpAddress":{"aws:SourceIp":["192.0.2.0/32"]}}}]}'

IAM-based Policy Example

You create IAM-based access policies by using the AWS IAM console rather than the Amazon ES console. For information about creating IAM-based access policies, see the IAM documentation.

Configuring Access Policies (AWS SDKs)

The AWS SDKs (except the Android and iOS SDKs) support all the actions defined in the Amazon ES Configuration API Reference, including the --access-policies parameter to the update-elasticsearch-domain-config command. For more information about installing and using the AWS SDKs, see AWS Software Development Kits.

Configuring Snapshots

Amazon Elasticsearch Service provides automatic daily snapshots of a domain's primary index shards and the number of replica shards. By default, the service takes automatic snapshots at midnight, but you can choose a different time.

Caution

The service stops taking snapshots of Amazon ES indices while the health of a cluster is RED. Subsequent uploads to indices in a RED cluster, even indices with a health status of GREEN, could be lost in the event of a cluster failure due to the cessation of snapshots. To prevent loss of data, return the health of your cluster to GREEN before uploading additional data to any index in the cluster.

Configuring Snapshots (Console)

Use the following procedure to configure daily automatic index snapshots by using the console.

To configure automatic snapshots

  1. Go to https://aws.amazon.com, and then choose Sign In to the Console.

  2. Under Analytics, choose Elasticsearch Service.

  3. In the navigation pane, under My domains, choose the domain that you want to update.

  4. Choose Configure cluster.

  5. For Automated snapshot start hour, choose the new hour for the service to take automated snapshots.

  6. Choose Submit.

Configuring Snapshots (AWS CLI)

Use the following syntax for the --snapshot-options option. The syntax for the option is the same for both the create-elasticsearch-domain and update-elasticsearch-domain-config commands.

Syntax

--snapshot-options AutomatedSnapshotStartHour=<value>

Parameter Valid Values Description
AutomatedSnapshotStartHour Integer between 0 and 23 Specifies the hour in UTC during which the service performs a daily automated snapshot of the indices in the new domain. The default value is 0, or midnight, which means that the snapshot is taken anytime between midnight and 1:00 AM.

Example

The following example configures automatic snapshots at 01:00 UTC:

Copy
aws es update-elasticsearch-domain-config --domain-name mylogs --region us-east-1 --snapshot-options AutomatedSnapshotStartHour=1

Configuring Snapshots (AWS SDKs)

The AWS SDKs (except the Android and iOS SDKs) support all the actions that are defined in the Amazon ES Configuration API Reference. This includes the --snapshots-options parameter to the update-elasticsearch-domain-config command. For more information about installing and using the AWS SDKs, see AWS Software Development Kits.

Configuring Advanced Options

Use advanced options to configure the following:

rest.action.multi.allow_explicit

Specifies whether explicit references to indices are allowed inside the body of HTTP requests. If you want to configure access policies for domain sub-resources, such as specific indices and domain APIs, you must disable this property. For more information, see URL-based access control. For more information about access policies for sub-resources, see Configuring Access Policies.

indices.fielddata.cache.size

Specifies the percentage of Java heap space that is allocated to field data. By default, this setting is unbounded.

Configuring Advanced Options (Console)

Use the following procedure to disable processing of HTTP requests with explicit references to indices in the request body. By default, the value is true.

To configure advanced options (console)

  1. Go to https://aws.amazon.com, and then choose Sign In to the Console.

  2. Under Analytics, choose Elasticsearch Service.

  3. In the navigation pane, under My domains, choose the domain that you want to update.

  4. Choose Configure cluster.

  5. Choose Advanced options.

  6. For rest.action.multi.allow_explicit_index, choose false to disable explicit references to indices in the bodies of HTTP requests.

  7. For indices.fielddata.cache.size, enter the percentage of Java heap space that is allocated to field data cache.

    By default, this setting is unbounded. For more information about the field data cache, see Field data in the Elasticsearch documentation.

    Note

    Many customers query rotating daily indices. We recommend that you begin benchmark testing with indices.fielddata.cache.size configured to 40% of the JVM heap for most such use cases. However, if you have very large indices you might need a larger field data cache.

  8. Choose Submit.

Configuring Advanced Options (AWS CLI)

Use the following syntax for the --advanced-options option. The syntax for the option is the same for both the create-elasticsearch-domain and update-elasticsearch-domain-config commands.

Syntax

--advanced-options rest.action.multi.allow_explicit=<true|false>, indices.fielddata.cache.size=<percentage_heap>

Parameter Valid Values Description
--advanced-options rest.action.multi.allow_explicit=<true|false>, Specifies whether explicit references to indices are allowed in an HTTP request body. Such references are always allowed in the HTTP request URL. Must be false when configuring access to individual sub-resources. By default, the value is true.
indices.fielddata.cache.size=<percentage_heap> Specifies the percentage of heap space to allocate to the field data cache. By default, this setting is unbounded.

Note

Many customers query rotating daily indices. We recommend that you begin benchmark testing with indices.fielddata.cache.size configured to 40% of the JVM heap for most such use cases. However, if you have very large indices you might need a large field data cache.

Example

The following example disables explicit references to indices in the HTTP request bodies and limits the field data cache to 40 percent of the total Java heap:

aws es update-elasticsearch-domain-config --domain-name mylogs --region us-east-1 --advanced-options rest.action.multi.allow_explicit_index=false, indices.fielddata.cache.size=40

Configuring Advanced Options (AWS SDKs)

The AWS SDKs (except the Android and iOS SDKs) support all of the actions defined in the Amazon ES Configuration API Reference, including the --advanced-options parameter to the update-elasticsearch-domain-config command. For more information about installing and using the AWS SDKs, see AWS Software Development Kits.