Tutorial: Using the open-source Elasticsearch Spark Connector - AWS Glue Studio

Tutorial: Using the open-source Elasticsearch Spark Connector

Elasticsearch is a popular open-source search and analytics engine for use cases such as log analytics, real-time application monitoring, and clickstream analysis. You can use Elasticsearch as a data store for your extract, transform, and load (ETL) jobs by configuring the Elasticsearch Spark Connector in AWS Glue Studio. This connector is available for free from AWS Marketplace.

In this tutorial, we will show how to connect to your Amazon Elasticsearch Service nodes with a minimal number of steps.

Prerequisites

To use this tutorial, you must have the following:

  • Access to AWS Glue Studio

  • Access to an Elasticsearch cluster in the AWS Cloud

  • Configured access to the Amazon VPC that contains your data store, as described in Configuring a VPC for your ETL job.

  • Configured permissions according to Job-related permissions

  • (Optional) Access to AWS Secrets Manager.