Architecture details - Centralized Logging with OpenSearch

Architecture details

This section describes the components and AWS services that make up this solution and the architecture details on how these components work together.

AWS services in this solution

The following AWS services are included in this solution:

AWS service Description

Amazon CloudFront

To distribute the frontend web UI assets.

Amazon S3

To store the static web assets (frontend user interface), and also uses it as a data buffer for log shipping.

Amazon Cognito

To authenticate users (in AWS Regions).

AWS AppSync

To provide the backend GraphQL APIs.

Amazon DynamoDB

To store the solution related information as a backend database.

AWS Lambda

To interact with other AWS services to process the core logic of managing log pipelines or log agents, and obtain information updated in DynamoDB tables.

AWS Step Functions

To orchestrate on-demand AWS CloudFormation deployment of a set of predefined stacks for log pipeline management.

AWS CloudFormation

To provision the AWS resources for the modules of pipelines and the solution web console.

AWS Systems Manager

To manage log agents for collecting logs from application servers, such as installing log agents (Fluent Bit) for application servers.

Amazon Kinesis Data Streams

To subscribe to logs from a CloudWatch Log Group or as a data buffer for log shipping, and then initiate the Log Processor Lambda to run.

Amazon Data Firehose

To subscribe the logs from CloudWatch Log Group and then put the logs into Amazon S3.

Amazon SQS

To receive Amazon S3 Event Notifications and then initiate the Log Processor Lambda to run.

Amazon Athena

To build the Light Engine.

AWS Glue

To build the Light Engine.