Migrate to the Centralized Logging with OpenSearch solution
Time to migrate: Approximately 60 minutes
In March 2023, a new AWS Solution,
Centralized
Logging with OpenSearch
This migration guide outlines steps to reuse the Amazon OpenSearch Service domain provisioned by the legacy solution in the new solution, so you do not lose historical log data. If you do not need the historical log data, consider deleting all resources provisioned by the legacy solution, and use the implementation guide to launch the new solution.
Note
Don't upgrade to the new solution if you want to deploy in AWS GovCloud (US) Regions. Instead, continue using the legacy solution.
Comparison between the two solutions
Features | Supported by new solution? | Supported by legacy solution? |
---|---|---|
Management | ||
Web Console | Yes | No |
Log source (AWS service logs) | ||
Amazon CloudTrail logs | Yes | Yes |
VPC Flow Logs | Yes | Yes |
Amazon S3 access logs | Yes | No |
AWS Config logs | Yes | No |
Amazon CloudFront standard logs | Yes | No |
Amazon CloudFront real-time logs | Yes | No |
Application Load Balancer logs | Yes | No |
AWS WAF logs | Yes | No |
Amazon RDS Aurora/MySQL logs | Yes | No |
AWS Lambda logs | Yes | No |
Log source (application logs) | ||
EC2 application logs | Yes | No |
EKS pod logs | Yes | No |
Syslog | Yes | No |
Log processor | ||
Predefined log parser - Apache | Yes | Yes |
Predefined log parser - Nginx | Yes | No |
Predefined log parser - Spring Boot | Yes | No |
Predefined pog parser - JSON | Yes | No |
Predefined log parser - Regex (Single line and multiple lines) | No | No |
Log enrichment - Geo to IP | Yes | No |
Log enrichment - User agent parser | Yes | No |
Log agent | ||
Amazon CloudWatch Logs agent | No | Yes |
FluentBit log agent | Yes | No |
Automatic installation and monitoring | Yes | No |
Automatic configuration | Yes | No |
Built-in OpenSearch dashboards | ||
Amazon CloudTrail logs | Yes | Yes |
VPC Flow Logs | Yes | Yes |
Amazon S3 access logs | Yes | No |
AWS Config logs | Yes | No |
Amazon CloudFront standard logs | Yes | No |
Amazon CloudFront real-time logs | Yes | No |
Application Load Balancer logs | Yes | No |
AWS WAF logs | Yes | No |
Amazon RDS Aurora/MySQL logs | Yes | No |
AWS Lambda logs | Yes | No |
Nginx access logs | Yes | No |
Apache HTTP Server access logs | Yes | No |
OpenSearch domain management | ||
Provision domain | No | Yes |
Import existing domain | Yes | No |
CloudWatch Alarms for domains | Yes | No |
Index lifecycle management | Yes | No |
Proxy stack (Nginx) | Yes | No |
Jumpbox server (Windows) | No | Yes |
Supported AWS Regions | ||
AWS Standard Regions | Yes | Yes |
AWS China Regions | Yes | No |
GovCloud Region | No | Yes |
Cross account and cross Region | ||
Cross-account log ingestion | Yes | Yes |
Cross-Region log ingestion | Yes | Yes |
Authentication option | ||
Amazon Cognito user pool | Yes | Yes |
OpenID Connect | Yes | No |
Network option | ||
Launch with new VPC | Yes | Yes |
Launch with existing VPC | Yes | No |
Migration overview
Both the
new
solution
Step 1: Deploy the new solution
Follow the Automated deployment steps to deploy the Centralized Logging with OpenSearch
solution. You can choose deployment with Amazon Cognito user pools or OpenID
Connect
Step 2. Upgrade the existing OpenSearch Service domains
The legacy solution provisioned an OpenSearch Service domain with engine version Elasticsearch 7.7. To work with the new solution, upgrade the OpenSearch Service domain to OpenSearch 1.0 and above. The upgrade process takes about 30 minutes.
-
Sign in to the Amazon OpenSearch Service console
. -
Under Domains, select the
centralizedlogging
OpenSearch Service domain.Note
centralizedlogging
is the fixed name of the OpenSearch Service domain provisioned by the legacy solution. -
Choose Actions on the top right corner, and select Upgrade.
-
Select OpenSearch 1.3 (latest) from the list.
-
Type upgrade in the text field, and choose Upgrade.
Step 3: Import the existing OpenSearch Service domain into the new solution
After the domain has completed upgrading in step 2, you can import it into the new solution.
-
Go to the AWS CloudFormation console
, and select the stack provisioned in Step 1: Deploy the new solution. -
Choose the Outputs tab, and then select the value for WebConsoleUrl. The URL opens in your web browser.
-
Input the credentials, and choose Sign In. The email is the one you used when provisioning the new solution, and a temporary password will be sent to the email address. When you sign in to the system for the first time, you will be asked to set a new password.
-
After signing in to the Centralized Logging with OpenSearch console, select Import OpenSearch Domain in the left navigation bar.
-
Follow the steps to Import an Amazon OpenSearch Service domain.
Step 4: Create new log analytics pipelines using the new solution
Use the new solution’s web console to ingest either AWS service logs or application logs. Follow the steps in the AWS Services Logs and Application Logs sections to start creating new log pipelines.
The new log analytic pipelines create and use new indices in the OpenSearch Service domain. You cannot migrate the existing data with the new pipelines. However, you can choose to retain existing data by keeping the corresponding indices as needed.
Step 5 (Optional): Delete provisioned resources by the legacy solution
The new solution can reuse the domains provisioned by the legacy solution. If you no longer need the legacy log analytic pipelines, go to CloudFormation console and delete the old stack. The OpenSearch Service domain and VPC will be retained after you delete the old stacks.
FAQ
Question: Can I migrate the existing data in Elasticsearch domain to work with the new pipeline?
Answer: No. This is not supported. You can keep your existing data (index) in your Elasticsearch domain.
Question: Do I have to delete the legacy solution from my AWS account when migrating to the new solution?
Answer: You can delete the stack of the legacy solution from the CloudFormation console if you no longer need old log analytics pipelines. Note that the Elasticsearch domain and VPC will be retained after you delete the old stacks and can be used in the new solution.
Question: Do I have to delete the Jumpbox server created in the legacy solution?
Answer: It depends. In the legacy solution, you can choose to deploy a Jumpbox server to access the Kibana dashboard within the VPC. In the new version, we have introduced the Nginx proxy stack, which is a new way to access OpenSearch Dashboards. The new way requires that you have a custom domain name and the SSL certificate.
Question: Can I reuse the Amazon Cognito user pool for OpenSearch Service to access the web console?
Answer: The Amazon Cognito user pool in the legacy solution is used to access OpenSearch Dashboards. In the new solution, we also provision an Amazon Cognito user pool for authentication and authorization for the frontend web console and backend APIs by default. If you want to use the existing user pool, you can launch the new solution with OpenID Connect.
Question: Can I deploy the new solution with the existing VPC created in the legacy solution?
Answer: You can deploy the new solution with either a new VPC or using the existing VPC. VPC peering is required if you want to access the Elasticsearch domain in the existing VPC. The new solution requires at least two private subnets with NAT gateway. By default, the VPC created by the old solution doesn’t have them, you might need to add NAT gateway to the VPC yourself.
Question: I have already used CloudWatch Logs agent to collect logs from Amazon EC2, do I need to migrate?
Answer: The new version uses Fluent Bit
Question: The new application log pipeline creates new Amazon Kinesis data streams, can I use the one created in the legacy solution?
Answer: No. In the legacy solution, we use Firehose to consume the messages from Kinesis Data Streams. However, in the new solution, we have replaced the Firehose with an AWS Lambda based custom log processor. You must create new pipelines from the web console, but to avoid additional cost, you should delete the old Kinesis data streams if they are no longer used.
Question: I already use the legacy solution to analyze VPC Flow Logs or AWS CloudTrail logs, do I need to migrate?
Answer: We recommend that you migrate to the new
solution, which supports both VPC Flow Logs and CloudTrail logs from Amazon Simple Storage Service
Question: How can I consume CloudWatch custom logs using the new solution?
Answer: You can use Firehose to subscribe CloudWatch Logs and transfer logs into Amazon S3. Complete the following steps to use the new solution to ingest logs from Amazon S3 to OpenSearch:
Create subscription filters with Firehose by following the instructions in Cross-account log data sharing using Kinesis Data Firehose.
Transfer your logs to Amazon S3 by following the instructions in Creating an Amazon Kinesis Data Firehose Delivery Stream.