Revisions - Centralized Logging with OpenSearch

Revisions

Date Changes
March 2023 Initial release.
April 2023

Released version 1.0.1

  • Fixed deployment failure due to Amazon S3 ACL changes.

June 2023

Released version 1.0.3

  • Fixed the EKS Fluent Bit deployment configuration generation issue.

August 2023

Released version 2.0.0

  • Added feature of ingesting log from S3 bucket continuously or on-demand

  • Added log pipeline monitoring dashboard into the solution console

  • Supported one-click enablement of pipeline alarms

  • Added an option to automatically attach required IAM policies when creating an Instance Group

  • Displayed an error message on the console when the installation of the log agent fails

  • Updated Application log pipeline creation process by allowing customer to specify a log source

  • Added validations to OpenSearch domain when importing a domain or selecting a domain to create a log pipeline

  • Supported installing log agent on AL2023 instances

  • Supported ingesting AWS WAF (associated with CloudFront) sampled logs to OpenSearch in other Regions except us-east-1

  • Allowed the same index name in different OpenSearch domains

November 2023

Released version 2.1.0

  • Added Light Engine to provide an Athena-based serverless and cost-effective log analytics engine to analyze infrequent access logs

  • Added OpenSearch Ingestion to provide more log processing capabilities, with which OSI can provision compute resource OpenSearch Compute Units (OCU) and pay per ingestion capacity

  • Supported parsing logs in nested JSON format

  • Supported CloudTrail logs ingestion from the specified bucket manually

  • Fixed the issue that the solution cannot list instances when creating instance groups

  • Fixed the issue that EC2 instances launched by the Auto Scaling group failed to pass the health check

December 2023

Released version 2.1.1

  • Fixed the issue that Log ingestion error in light engine when not specified time key in the log config

  • Fixed the issue that cannot deploy the Centralized Logging with OpenSearch solution in the UAE Region

  • Fixed the issue that EC2 instances should not be added to the same Instance Group

March 2024

Released version 2.1.2

  • Resolved a bug where upgrading from versions earlier than 2.1.0 led to the loss of Amazon S3 notifications, preventing the proper collection of logs from the Amazon S3 buffer.

  • Addressed a problem where including the @timestamp field in log configurations caused failures in creating index_templates, leading to an inability to write data to OpenSearch.

  • Fixed a bug in the log processor Lambda due to the absence of the 'batch_size' variable, causing process failures.

  • Solved a deployment issue with the Log Analytics Pipeline, which previously could not deploy cross-account Lambda pipelines.

  • Corrected an issue with the Application Load Balancer Service Log Parser that resulted in the omission of numerous log lines.

  • Amended an inaccurate warning message displayed during pipeline creation with an existing index in OpenSearch.

  • Amended an inaccurate error messaging when deleting an Instance Group in application log pipelines.

June 2024

Released version 2.2.0

  • Added support to install, configure, and monitor the Fluent Bit agent on Windows Server 2016/2019/2022 instances. 

  • Added a log visualization template for Windows IIS (W3C format) logs.

  • Added a default tag "CLOSolutionCostAnalysis" to all resources created by the solution, allowing customers to view the total cost in Cost Explorer.

  • Add support for tag propagation for resources when creating log pipelines, allowing customers to view all resources and associated costs at the pipeline level.

  • Added support for ingesting and analyzing data in a specified Amazon S3 location using Light Engine.

  • Added support for importing OpenSearch clusters with custom KMS encryption enabled.

  • Added support for ingesting and analyzing AWS CloudTrail logs using Light Engine.

  • Added support for ingesting and analyzing Syslog data using Light Engine.

  • Added support for using the Unix timestamp as the time key in Log Config.

  • Added support for Asia Pacific (Hyderabad), Asia Pacific (Jakarta), Asia Pacific (Melbourne), Israel (Tel Aviv), Canada (Calgary), Europe (Spain), Europe(Zurich), Middle East (UAE) Regions.

  • Fixed an issue where creating a pipeline failed when Lambda reached concurrency limits.

  • Fixed a bug where the system could not read properties of undefined ('accountId') when the Next button was clicked without selecting an Instance Group.

  • Fixed an issue where logs were not received when using the solution-provisioned staging bucket in Light Engine.

  • Fixed a permissions issue in the LogMerger State Machine within Light Engine: The S3ObjectMigration Lambda failed due to insufficient AWS KMS permissions on the analytics S3 bucket.