Architecture overview

Deploying this solution with the default parameters builds the following environment in the AWS Cloud.

Figure 1: Centralized Logging solution architecture on AWS

The Centralized Logging solution contains the following components: log ingestion, log indexing, and visualization.

Note

Deploy the AWS CloudFormation template in the AWS account where you intend to store your log data.

Log ingestion

For the log ingestion component, the AWS CloudFormation template deploys Amazon CloudWatch Logs destinations in the primary account.

Figure 2: Centralized Logging log ingestion component

1. This solution uses the CloudWatch Logs destination capability for log streaming. CloudWatch Logs destinations are created with the required permissions in each of the selected Regions in your primary account. After the destinations are created with the necessary permissions, you can configure CloudWatch Logs subscription filters for log groups to be streamed to the centralized logging account. For information about creating custom CloudWatch Logs, refer to Adding custom CloudWatch Logs.

   Note

   You can control the spoke streams that will log events to the primary account using the Spoke Account and Spoke Region parameters. These parameters can be updated at any time after installation to add/remove accounts and Regions.

2. An optional demo AWS CloudFormation template can be deployed to generate sample CloudWatch Logs for AWS CloudTrail, Amazon Virtual Private Cloud (Amazon VPC) flow logs, and an Amazon Elastic Compute Cloud (Amazon EC2) web server. For information about the sample logs, refer to Sample logs. The demo template configures each of these log groups with the needed subscription filters to stream log events to the CloudWatch Logs destination in the centralized logging account, as shown in Figure 2.

   Important

   Since the sample logs Apache web server is publicly accessible, we do not recommend deploying the demo AWS CloudFormation template in a production environment.

Log indexing

For the log indexing component, the AWS CloudFormation template deploys Amazon Kinesis Data Streams, AWS Lambda functions, Amazon Kinesis Data Firehose, and Amazon OpenSearch Service, as shown in Figure 3.

Figure 3: Centralized Logging log indexing component

1. A centralized Kinesis Data Streams is provisioned to index log events on the centralized Amazon OpenSearch Service domain. The CloudWatch Logs destinations created to stream log events, have Kinesis Data Streams as their target.

2. Once the log events stream to Kinesis Data Streams, the service invokes an AWS Lambda function to transform each log event to an Amazon OpenSearch Service document, which is then put in Kinesis Data Firehose.

3. Kinesis Data Firehose indexes the documents on the Amazon OpenSearch Service domain.

4. Kinesis Data Firehose logs errors in CloudWatch and delivers the records to Amazon Simple Storage Service (Amazon S3) for low-cost storage.

   Note

   You can monitor Kinesis Data Firehose as it sends custom CloudWatch Logs containing detailed monitoring data for each delivery stream.

Visualization

This solution provides data visualization and exploration support using Amazon OpenSearch Service and Kibana. An Amazon OpenSearch Service domain is created inside an Amazon VPC, preventing public access to the Kibana dashboard. Access to the Kibana dashboard is secured using a VPC security group and an AWS Identity and Access Management (IAM) role.

Figure 4: Centralized Logging visualization component

This solution optionally launches a Microsoft Windows Jumpbox Server that can be used to access the Amazon OpenSearch Service cluster and Kibana dashboard. An administrator account is configured to provide the permission to access the Kibana dashboard. This solution uses an Amazon Cognito user pool and an identity pool for authentication and authorization. For additional information about this solution’s security, refer to Security.