Migrate an ELK Stack to Elastic Cloud on AWS - AWS Prescriptive Guidance

Migrate an ELK Stack to Elastic Cloud on AWS

Created by Battulga Purevragchaa (AWS), Uday Theepireddy (Elastic), and Antony Prasad Thevaraj (AWS)

Environment: Production

Source: Elasticsearch

Target: Elastic Cloud

R Type: Replatform

Workload: All other workloads

Technologies: Analytics; Security, identity, compliance

AWS services: Amazon EC2; Amazon EC2 Auto Scaling; Elastic Load Balancing; Amazon S3; Amazon Route 53

Summary

Elastic has provided services for many years, with their users and customers typically managing Elastic themselves on premises. Elastic Cloud, the managed Elasticsearch service, provides a way to consume the Elastic Stack (ELK Stack) and solutions for enterprise search, observability, and security. You can access Elastic solutions with apps such as Logs, Metrics, APM (application performance monitoring), and SIEM (security information and event management). You can use integrated features such as machine learning, index lifecycle management, Kibana Lens (for drag-and drop visualizations).

When you move from self-managed Elasticsearch to Elastic Cloud, the Elasticsearch service takes care of the following:

  • Provisioning and managing the underlying infrastructure

  • Creating and managing Elasticsearch clusters

  • Scaling clusters up and down

  • Upgrades, patching, and taking snapshots

This gives you more time to focus on solving other challenges.

This pattern defines how to migrate on-premises Elasticsearch 7.13  to Elasticsearch on Elastic Cloud on Amazon Web Services (AWS). Other versions might require slight modifications to the processes described in thie pattern. For more information, contact your Elastic representative.

Prerequisites and limitations

Prerequisites

You must understand how large your snapshots and the  lifecyle policies for accompanying indexes are on premises before initiating your migration. For more information, contact Elastic.

Roles and skills

The migration process also requires the roles and expertise described in the following table.

RoleExpertiseResponsibilities
App supportFamiliarity with Elastic Cloud and Elastic on premisesAll Elastic related tasks
Systems administrator or DBAIn-depth knowledge of the on-premises Elastic environment and its configurationThe ability to provision storage, install and use the AWS Command Line Interface (AWS CLI), and identify all data sources feeding Elastic on premises
Network administratorKnowledge of on-premises to AWS network connectivity, security, and performanceEstablishment of network links from on premises to Amazon S3, with an understanding of connectivity bandwidth

Limitations 

Product versions

  • Elasticsearch 7.13

Architecture

Source technology stack  

On-premises Elasticsearch 7.13 or later:

  • Cluster snapshots

  • Index snapshots

  • Beats configuration

Source technology architecture

The following diagram shows a typical on-premises architecture with different ingestion methods, node types, and Kibana. The different node types reflect the Elasticsearch cluster, authentication, and visualization roles.

  1. Ingestion from Beats to Logstash

  2. Ingestion from Beats to Apache Kafka messaging queue

  3. Ingestion from Filebeat to Logstash

  4. Ingestion from Apache Kafka messaging queue to Logstash

  5. Ingestion from Logstash to an Elasticsearch cluster

  6. Elasticsearch cluster

  7. Authentication and notification node

  8. Kibana and blob nodes

Target technology stack 

Elastic Cloud is deployed to your software as a service (SaaS) account in multiple AWS Regions with cross-cluster replication.

  • Cluster snapshots

  • Index snapshots

  • Beats configurations

  • Elastic Cloud

  • Network Load Balancer

  • Amazon Route 53

  • Amazon S3

Target architecture 

The managed Elastic Cloud infrastructure is:

High-level migration steps

Elastic has developed its own prescriptive methodology for migrating on-premises Elastic Cluster to Elastic Cloud. The Elastic methodology is directly aligned and complementary to the AWS migration guidance and best practices, including Well-Architected Framework and AWS Migration Acceleration Program (MAP). Typically, the three AWS migration phases are the following:

  • Assess

  • Mobilize

  • Migrate and modernize

Elastic follows similar migration phases with complementary terminology:

  • Initiate

  • Plan

  • Implement

  • Deliver

  • Close

Elastic uses the Elastic Implementation Methodology to facilitate the delivery of project outcomes. This is inclusive by design to ensure that the Elastic, consulting teams, and customer teams work together with clarity to jointly deliver intended outcomes.

The Elastic methodology combines traditional waterfall phasing with Scrum within the implementation phase. Configurations of technical requirements are delivered iteratively in a collaborative manner while minimizing risk.

Tools

  • Elastic Load Balancing – Elastic Load Balancing automatically distributes your incoming traffic across multiple targets, such as EC2 instances, containers, and IP addresses, in one or more Availability Zones.

  • Amazon Route 53 – Amazon Route 53 is a highly available and scalable Domain Name System (DNS) web service. You can use Route 53 to perform three main functions in any combination: domain registration, DNS routing, and health checking.

  • Amazon S3 – Amazon Simple Storage Service (Amazon S3) is an object storage service. You can use Amazon S3 to store and retrieve any amount of data at any time, from anywhere on the web. This pattern uses an S3 bucket and Amazon S3 Transfer Acceleration.

  • Beats – Beats ship data from Logstash or Elasticsearch

  • Elastic Cloud – Elastic Cloud is a managed service for hosting Elasticsearch.

  • Elasticsearch – Elasticsearch is a search and analytics engine that uses the Elastic Stack to centrally store your data for search and analytics that scale. This pattern also uses snapshot creation and cross-cluster replication.

  • Logstash – Logstash is a server-side data processing pipeline that ingests data from multiple sources, transforms it, and then sends it to your data storage.

Epics

TaskDescriptionSkills required
Identify servers running the on-premises Elastic solution.

Confirm that Elastic migration is supported.

App owner
Understand the on-premises server configuration.

To understand the server configuration needed to drive workloads successfully on premises, find the server hardware footprint, network configuration, and storage characteristics that are currently in use

App Support
Gather user and app account information.

Identify the user names and app names that are used by the on-premises Elastic environment.

Systems administrator, App support
Document Beats and data shipper configuration.

To document the configurations, look at existing data sources and sinks. For more information, see the Elastic documentation.

App support
Determine the velocity and volume of data.

Establish a baseline for how much data the cluster is handling.

Systems administrator, App support
Document RPO and RTO scenarios.

Document recovery point objective (RPO) and recovery time objective (RTO) scenarios in terms of outages and service level agreements (SLAs).

App owner, Systems administrator, App support
Determine the optimal snapshot lifecycle settings.

Define how often data needs to be secured by using Elastic snapshots during and after the migration.

App owner, Systems administrator, App support
Define post-migration performance expectations.

Generate metrics on current and expected screen refresh, query runtimes, and user interface behaviors.

Systems administrator, App support
Document internet access transport, bandwidth, and availability requirements.

Ascertain speed, latency, and resiliency of internet connections for copying snapshots to Amazon S3.

Network administrator
Document current costs of on-premises runtime for Elastic.

Ensure that the sizing of the AWS targeted environment is designed to be both high performing and cost effective.

DBA, Systems administrator, App support
Identify the authentication and authorization needs.

The Elastic Stack security features provide built-in realms such as Lightweight Directory Access Protocol (LDAP), Security Assertion Markup Language (SAML), and OpenID Connect (OIDC).

DBA, Systems administrator, App support
Understand the specific regulatory requirements based on the geographic location.

Ensure that data is exported and encrypted according to your requirements and to any relevant national requirements.

DBA, Systems administrator, App support
TaskDescriptionSkills required
Prepare the staging area on Amazon S3.

To receive snapshots on Amazon S3, set up the Amazon S3 area by following the steps in the Elasticsearch documentation. Use the es-s3-Snapshot customer managed policy attached to a new es-Snapshot-user AWS Identity and Access Management (IAM) account. Keep the access key ID and secret access key secured.

Enable Amazon S3 Transfer Acceleration on the bucket.

AWS administrator
Install AWS CLI and the Amazon S3 plugin on premises.

On each Elasticsearch node, run the following command.

sudo bin/elasticsearch-plugin install repository-s3

Then reboot the node.

AWS administrator
Configure Amazon S3 client access.

Use the keys from the IAM user es-Snapshot-user by running the following commands.

elasticsearch-keystore add s3.client.default.access_key
elasticsearch-keystore add s3.client.default.secret_key
AWS administrator
Register a snapshot repository for Elastic data

Use the Kibana Dev Tools to tell the on-premises local cluster which remote S3 bucket to write to.

AWS administrator
Configure snapshot policy.

To configure snapshot lifecycle management, on the Kibana Policies tab, choose SLM policy, and define which times, data streams, or indexes should be included, and what names to use.

Configure a policy that takes frequent snapshots. Snapshots are incremental and make efficient use of storage. Match your readiness assessment decision. A policy can also specify a retention policy and automatically delete snapshots when they are no longer needed.

App support
Verify that snapshots work.

In Kibana Dev Tools, run the following command.

GET _snapshot/<your_repo_name>/_all
AWS administrator, App support,
Deploy a new cluster on Elastic Cloud.

Log in to Elastic, and choose a cluster for “observability, search or security” derived from your business findings in the readiness assessment.

AWS administrator, App support
Set up cluster key store access.

The new cluster needs access to the S3 bucket that will store the snapshots. On the Elasticsearch Service Console, choose Security, and enter the access and secret IAM keys that you created earlier.

AWS administrator
Configure the Elastic Cloud hosted cluster to access Amazon S3.

Set up new cluster access to the previously created snapshot repository in Amazon S3. Using Kibana, do the following:

  1. Choose Stack Management, Snapshot Settings, RegisterRepo.

  2. In the Alias field, enter the name of the repository.

  3. For S3 Client name, choose secondary.

  4. Add the S3 bucket that you created earlier to the repository.

  5. Choose Compress snapshot.

  6. For the Encryption settings, keep the default values.

AWS administrator, App Support
Verify the new Amazon S3 repository.

Ensure that you can access your new repository hosted in the Elastic Cloud cluster.

AWS administrator
Initilaize the Elasticsearch service cluster.

On the Elasticsearch Service Console, initialize the Elasticsearch service cluster from the S3 snapshot.

Run the following commands as POST.

*/_close?expand_wildcards=all
/_snapshot/<your-repo-name>/  <your-snapshot-name>/_restore
*/_open?expand_wildcards=all
App Support
TaskDescriptionSkills required
Verify that the snapshot restore was successful.

Using Kibana Dev Tools, run the following command.

GET _cat/indices
App support
Redploy ingestion services.

Connect the endpoints for Beats and Logstash to the new Elasticsearch service endpoint.

App support
TaskDescriptionSkills required
Validate the cluster environment.

After the on-premises Elastic cluster environment is migrated to AWS, you can connect to it and use your own user acceptance testing (UAT) tools to validate the new environment.

App support

Related resources

Elastic references

Elastic blog posts

Elastic documentation

Elastic video and webinar

AWS references

Additional information

If you're planning to migrate complex workloads, engage Elastic Consulting Services. If you have basic questions related to configurations and services, contact the Elastic Support team.