Tutorial: Amazon Elasticsearch Service Resolvers - AWS AppSync

Tutorial: Amazon Elasticsearch Service Resolvers

AWS AppSync supports using Amazon Elasticsearch Service from domains that you have provisioned in your own AWS account, provided they don’t exist inside a VPC. After your domains are provisioned, you can connect to them using a data source, at which point you can configure a resolver in the schema to perform GraphQL operations such as queries, mutations, and subscriptions. This tutorial will take you through some common examples.

For more information, see the Resolver Mapping Template Reference for Elasticsearch.

One-Click Setup

To automatically setup a GraphQL endpoint in AWS AppSync with Amazon Elasticsearch Service configured you can use this AWS CloudFormation template:

After the AWS CloudFormation deployment completes you can skip directly to running GraphQL queries and mutations.

Create a New Amazon ES Domain

To get started with this tutorial, you need an existing Amazon ES domain. If you don’t have one, you can use the following sample. Note that it can take up to 15 minutes for an Amazon ES domain to be created before you can move on to integrating it with an AWS AppSync data source.

aws cloudformation create-stack --stack-name AppSyncElasticsearch \ --template-url https://s3.us-west-2.amazonaws.com/awsappsync/resources/elasticsearch/ESResolverCFTemplate.yaml \ --parameters ParameterKey=ESDomainName,ParameterValue=ddtestdomain ParameterKey=Tier,ParameterValue=development \ --capabilities CAPABILITY_NAMED_IAM

You can launch the following AWS CloudFormation stack in the US West 2 (Oregon) region in your AWS account:

Configure Data Source for Amazon ES

After the Amazon ES domain is created, navigate to your AWS AppSync GraphQL API and choose the Data Sources tab. Choose New and enter a friendly name for the data source, such as “Elasticsearch”. Then choose Amazon Elasticsearch domain for Data source type, choose the appropriate region, and you should see your Amazon ES domain listed. After selecting it you can either create a new role and AWS AppSync will assign the role-appropriate permissions, or you can choose an existing role, which has the following inline policy:

{ "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1234234", "Effect": "Allow", "Action": [ "es:ESHttpDelete", "es:ESHttpHead", "es:ESHttpGet", "es:ESHttpPost", "es:ESHttpPut" ], "Resource": [ "arn:aws:es:REGION:ACCOUNTNUMBER:domain/democluster/*" ] } ] }

You’ll also need to set up a trust relationship with AWS AppSync for that role:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "appsync.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }

Additionally, the Amazon ES domain has it’s own Access Policy which you can modify through the Amazon Elasticsearch Service console. You will need to add a policy similar to the below, with the appropriate actions and resource for the Amazon ES domain. Note that the Principal will be the AppSync data source role, which if you let the console create this would start with the name of appsync-datasource-es- and can be found in the AWS IAM console.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::ACCOUNTNUMBER:role/service-role/APPSYNC_DATASOURCE_ROLE" }, "Action": [ "es:ESHttpDelete", "es:ESHttpHead", "es:ESHttpGet", "es:ESHttpPost", "es:ESHttpPut" ], "Resource": "arn:aws:es:REGION:ACCOUNTNUMBER:domain/DOMAIN_NAME/*" } ] }

Connecting a Resolver

Now that the data source is connected to your Amazon ES domain, you can connect it to your GraphQL schema with a resolver, as shown in the following example:

schema { query: Query mutation: Mutation } type Query { getPost(id: ID!): Post allPosts: [Post] } type Mutation { addPost(id: ID!, author: String, title: String, url: String, ups: Int, downs: Int, content: String): Post } type Post { id: ID! author: String title: String url: String ups: Int downs: Int content: String } ...

Note that there is a user-defined Post type with a field of id. In the following examples, we assume there is a process (which can be automated) for putting this type into your Amazon ES domain, which would map to a path root of /id/post, where id is the index and post is the type. From this root path, you can perform individual document searches, wildcard searches with /id/post* or multi-document searches with a path of /id/post/_search. If you have another type User, for example, one that is indexed under the same index id, you can perform multi-document searches with a path of /id/_search. This searches for fields on both Post and User.

From the schema editor in the AWS AppSync console, modify the preceding Posts schema to include a searchPosts query:

type Query { getPost(id: ID!): Post allPosts: [Post] searchPosts: [Post] }

Save the schema. On the right side, for searchPosts, choose Attach resolver. Choose your Amazon ES data source. Under the request mapping template section, select the drop-down for Query posts to get a base template. Modify the path to be /id/post/_search. It should look like the following:

{ "version":"2017-02-28", "operation":"GET", "path":"/id/post/_search", "params":{ "headers":{}, "queryString":{}, "body":{ "from":0, "size":50 } } }

This assumes that the preceding schema has documents with an id field, and that the documents have been indexed in Amazon ES by this field. If you structure your data differently, then you’ll need to update accordingly.

Under the response mapping template section, you need to specify the appropriate _source filter if you want to get back the data results from an Amazon ES query and translate to GraphQL. Use the following template:

[ #foreach($entry in $context.result.hits.hits) #if( $velocityCount > 1 ) , #end $utils.toJson($entry.get("_source")) #end ]

Modifying Your Searches

The preceding request mapping template performs a simple query for all records. Suppose you want to search by a specific author. Further, suppose you want that author to be an argument defined in your GraphQL query. In the schema editor of the AWS AppSync console, add an allPostsByAuthor query:

type Query { getPost(id: ID!): Post allPosts: [Post] allPostsByAuthor(author: String!): [Post] searchPosts: [Post] }

Now choose Attach resolver and select the Amazon ES data source, but use the following example in the response mapping template:

{ "version":"2017-02-28", "operation":"GET", "path":"/id/post/_search", "params":{ "headers":{}, "queryString":{}, "body":{ "from":0, "size":50, "query":{ "term" :{ "author": $util.toJson($context.arguments.author) } } } } }

Note that the body is populated with a term query for the author field, which is passed through from the client as an argument. You could optionally have prepopulated information, such as standard text, or even use other utilities.

If you’re using this resolver, fill in the response mapping template with the same information as the previous example.

Adding Data to Amazon ES

You may want to add data to your Amazon ES domain as the result of a GraphQL mutation. This is a powerful mechanism for searching and other purposes. Because you can use GraphQL subscriptions to make your data real-time, it serves as a mechanism for notifying clients of updates to data in your Amazon ES domain.

Return to the Schema page in the AWS AppSync console and select Attach resolver for the addPost() mutation. Select the Amazon ES data source again and use the following response mapping template for the Posts schema:

{ "version":"2017-02-28", "operation":"PUT", "path": $util.toJson("/id/post/$context.arguments.id"), "params":{ "headers":{}, "queryString":{}, "body":{ "id": $util.toJson($context.arguments.id), "author": $util.toJson($context.arguments.author), "ups": $util.toJson($context.arguments.ups), "downs": $util.toJson($context.arguments.downs), "url": $util.toJson($context.arguments.url), "content": $util.toJson($context.arguments.content), "title": $util.toJson($context.arguments.title) } } }

As before, this is an example of how your data might be structured. If you have different field names or indexes, you need to update the path and body as appropriate. This example also shows how to use $context.arguments to populate the template from your GraphQL mutation arguments.

Before moving on, use the following response mapping template, which will be explained more in the next section:


Retrieving a Single Document

Finally, if you want to use the getPost(id:ID) query in your schema to return an individual document, find this query in the schema editor of the AWS AppSync console and choose Attach resolver. Select the Amazon ES data source again and use the following mapping template:

{ "version":"2017-02-28", "operation":"GET", "path": $util.toJson("/id/post/$context.arguments.id"), "params":{ "headers":{}, "queryString":{}, "body":{} } }

Because the path above uses the id argument with an empty body, this returns the single document. However, you need to use the following response mapping template, because now you’re returning a single item and not a list:


Perform Queries and Mutations

You should now be able to perform GraphQL operations against your Amazon ES domain. Navigate to the Queries tab of the AWS AppSync console and add a new record:

mutation { addPost( id:"12345" author: "Fred" title: "My first book" content: "This will be fun to write!" ){ id author title } }

If the record is inserted successfully, you’ll see the fields on the right. Similarly, you can now run a searchPosts query against your Amazon ES domain:

query { searchPosts { id title author content } }

Best Practices

  • Amazon ES should be for querying data, not as your primary database. You may want to use Amazon ES in conjunction with Amazon DynamoDB as outlined in Combining GraphQL Resolvers.

  • Only give access to your domain by allowing the AWS AppSync service role to access the cluster.

  • You can start small in development, with the lowest-cost cluster, and then move to a larger cluster with high availability (HA) as you move into production.