Creating a search application with Amazon OpenSearch Service - Amazon OpenSearch Service (successor to Amazon Elasticsearch Service)

Creating a search application with Amazon OpenSearch Service

A common way to create a search application with Amazon OpenSearch Service is to use web forms to send user queries to a server. Then you can authorize the server to call the OpenSearch APIs directly and have the server send requests to OpenSearch Service.

If you want to write client-side code that doesn't rely on a server, however, you should compensate for the security and performance risks. Allowing unsigned, public access to the OpenSearch APIs is inadvisable. Users might access unsecured endpoints or impact cluster performance through overly broad queries (or too many queries).

This chapter presents a solution: use Amazon API Gateway to restrict users to a subset of the OpenSearch APIs and AWS Lambda to sign requests from API Gateway to OpenSearch Service.

Note

Standard API Gateway and Lambda pricing applies, but within the limited usage of this tutorial, costs should be negligible.

Step 1: Index sample data

A prerequisite for these steps is an OpenSearch Service domain. Download sample-movies.zip, unzip it, and use the _bulk API to add the 5,000 documents to the movies index:

POST https://search-my-domain.us-west-1.es.amazonaws.com/_bulk { "index": { "_index": "movies", "_type": "movie", "_id": "tt1979320" } } {"directors":["Ron Howard"],"release_date":"2013-09-02T00:00:00Z","rating":8.3,"genres":["Action","Biography","Drama","Sport"],"image_url":"http://ia.media-imdb.com/images/M/MV5BMTQyMDE0MTY0OV5BMl5BanBnXkFtZTcwMjI2OTI0OQ@@._V1_SX400_.jpg","plot":"A re-creation of the merciless 1970s rivalry between Formula One rivals James Hunt and Niki Lauda.","title":"Rush","rank":2,"running_time_secs":7380,"actors":["Daniel Brühl","Chris Hemsworth","Olivia Wilde"],"year":2013,"id":"tt1979320","type":"add"} { "index": { "_index": "movies", "_type": "movie", "_id": "tt1951264" } } {"directors":["Francis Lawrence"],"release_date":"2013-11-11T00:00:00Z","genres":["Action","Adventure","Sci-Fi","Thriller"],"image_url":"http://ia.media-imdb.com/images/M/MV5BMTAyMjQ3OTAxMzNeQTJeQWpwZ15BbWU4MDU0NzA1MzAx._V1_SX400_.jpg","plot":"Katniss Everdeen and Peeta Mellark become targets of the Capitol after their victory in the 74th Hunger Games sparks a rebellion in the Districts of Panem.","title":"The Hunger Games: Catching Fire","rank":4,"running_time_secs":8760,"actors":["Jennifer Lawrence","Josh Hutcherson","Liam Hemsworth"],"year":2013,"id":"tt1951264","type":"add"} ...

To learn more, see Indexing data in Amazon OpenSearch Service.

Step 2: Create the API in API Gateway

Using API Gateway lets you create a more limited API and simplifies the process of interacting with the OpenSearch _search API. API Gateway lets you enable security features like Amazon Cognito authentication and request throttling. Perform the following steps to create and deploy an API:

Create and configure the API

To create your API using the API Gateway console

  1. Within API Gateway, choose Create API.

  2. Locate REST API (not private) and choose Build.

  3. Configure the following fields:

    • API name: opensearch-api

    • Description: Public API for searching an Amazon OpenSearch Service domain

    • Endpoint Type: Regional

  4. Choose Create API.

  5. Choose Actions and Create Method.

  6. Select GET in the dropdown and click the checkmark to confirm.

  7. Configure the following settings, then choose Save:

Setting Value
Integration type Lambda function
Use Lambda proxy integration Yes
Lambda region us-west-1
Lambda function opensearch-lambda (you'll configure this later in Lambda)
Use default timeout Yes

Configure the method request

Choose Method Request and configure the following settings:

Setting Value
Authorization NONE
Request Validator

Validate query string parameters and headers

API Key Required false

URL Query String Parameters

Setting Value
Name q
Required

Yes

Deploy the API and configure a stage

The API Gateway console lets you deploy an API by creating a deployment and associating it with a new or existing stage.

  1. Choose Actions and Deploy API.

  2. For Deployment stage choose New Stage and name the stage opensearch-api-test.

  3. Choose Deploy.

  4. Configure the following settings in the stage editor, then choose Save Changes:

Setting Value
Enable throttling Yes
Rate

1000

Burst 500

These settings configure an API that has only one method: a GET request to the endpoint root (https://some-id.execute-api.us-west-1.amazonaws.com/search-es-api-test). The request requires a single parameter (q), the query string to search for. When called, the method passes the request to Lambda, which runs the opensearch-lambda function. For more information, see Creating an API in Amazon API Gateway and Deploying a REST API in Amazon API Gateway.

Step 3: Create and deploy the Lambda function

After you create your API in API Gateway, create the Lambda function that it passes requests to.

Create the Lambda function

In this solution, API Gateway passes requests to the following Python 3.8 Lambda function, which queries OpenSearch Service and returns results. Name the function opensearch-lambda.

Because this sample function uses external libraries, you need to create a deployment package and upload it to Lambda for the code to work. For more information about creating Lambda functions and deployment packages, see Deploy Python Lambda functions with .zip file archives in the AWS Lambda Developer Guide and Create the Lambda deployment package in this guide.

import boto3 import json import requests from requests_aws4auth import AWS4Auth region = '' # For example, us-west-1 service = 'opensearchservice' credentials = boto3.Session().get_credentials() awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token) host = '' # The OpenSearch domain endpoint with https:// index = 'movies' url = host + '/' + index + '/_search' # Lambda execution starts here def lambda_handler(event, context): # Put the user query into the query DSL for more accurate search results. # Note that certain fields are boosted (^). query = { "size": 25, "query": { "multi_match": { "query": event['queryStringParameters']['q'], "fields": ["title^4", "plot^2", "actors", "directors"] } } } # Elasticsearch 6.x requires an explicit Content-Type header headers = { "Content-Type": "application/json" } # Make the signed HTTP request r = requests.get(url, auth=awsauth, headers=headers, data=json.dumps(query)) # Create the response and add some extra content to support CORS response = { "statusCode": 200, "headers": { "Access-Control-Allow-Origin": '*' }, "isBase64Encoded": False } # Add the search results to the response response['body'] = r.text return response

Modify the handler

The handler is the method in your function code that processes events. You need to change the handler name according to the name of the file in your deployment package where the Lambda function is located. For example, if your file is named function.py, rename the handler to function.lambda_handler. For more information, see Lambda function handler in Python.

Configure a trigger

Choose Add trigger and create the HTTP endpoint that invokes your function. The trigger must have the following configuration:

Trigger API Deployment Stage Security
API Gateway opensearch-api opensearch-api-test Open

Step 4: Modify the domain access policy

Your OpenSearch Service domain must allow the Lambda function to make GET requests to the movies index. The following policy provides opensearch-lambda-role (created through Lambda) access to the movies index:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::123456789012:role/service-role/opensearch-lambda-role-1abcdefg" }, "Action": "es:ESHttpGet", "Resource": "arn:aws:es:us-west-1:123456789012:domain/domain-name/movies/_search" } ] }
Note

To get the exact name of the role that Lambda automatically creates, go to the AWS Identity and Access Management (IAM) console, choose Roles, and search for "lambda".

For more information about access policies, see Configuring access policies.

Step 5: Test the web application

To test the web application

  1. Download sample-site.zip, unzip it, and open scripts/search.js in your favorite text editor.

  2. Update the apigatewayendpoint variable to point to your API Gateway endpoint. The endpoint takes the form of https://some-id.execute-api.us-west-1.amazonaws.com/opensearch-api-test.

  3. Open index.html and try running searches for thor, house, and a few other terms.

    
                        A sample search for thor.

Next steps

This chapter is just a starting point to demonstrate a concept. You might consider the following modifications:

  • Add your own data to the OpenSearch Service domain.

  • Add methods to your API.

  • In the Lambda function, modify the search query or boost different fields.

  • Style the results differently or modify search.js to display different fields to the user.