Architecture Overview - Centralized Logging

Architecture Overview

Deploying this solution with the default parameters builds the following environment in the AWS Cloud.


        Centralized Logging solution architecture on AWS

Figure 1: Centralized Logging solution architecture on AWS

The Centralized Logging solution contains the following components: log ingestion, log indexing, and visualization.

Note

Deploy the AWS CloudFormation template in the AWS account where you intend to store your log data.

Log ingestion

For the log ingestion component, the AWS CloudFormation template deploys Amazon CloudWatch Logs destinations in the primary account.


          Log ingestion component in the Centralized Logging solution

Figure 2: Log ingestion component in the Centralized Logging solution

This solution uses the CloudWatch Logs destination capability for log streaming. CloudWatch Logs destinations are created with the required permissions in each of the selected Regions in your primary account. After the destinations are created with the necessary permissions, you can configure CloudWatch Logs subscription filters for log groups to be streamed to the centralized logging account. For information about creating custom CloudWatch Logs, refer to Appendix C.

Note

You can control the spoke streams that will log events to the primary account using the Spoke Account and Spoke Region parameters. These parameters can be updated at any time after installation to add/remove accounts and Regions.

An optional demo AWS CloudFormation template can be deployed to generate sample CloudWatch Logs for AWS CloudTrail, Amazon Virtual Private Cloud (Amazon VPC) flow logs, and an Amazon Elastic Compute Cloud (Amazon EC2) web server. For information about the sample logs, refer to Appendix A. The demo template configures each of these log groups with the needed subscription filters to stream log events to the CloudWatch Logs destination in the centralized logging account, as shown in Figure 2.

Important

Since the sample logs Apache web server is publicly accessible, we do not recommend deploying the demo AWS CloudFormation template in a production environment.

Log indexing

For the log indexing component, the AWS CloudFormation template deploys Amazon Kinesis Data Streams, AWS Lambda functions, Amazon Kinesis Data Firehose, and Amazon ES, as shown in Figure 3.


          Log indexing in the Centralized Logging solution

Figure 3: Log indexing in Centralized Logging

  1. A centralized Kinesis Data Streams and Kinesis Data Firehose are provisioned to index log events on the centralized Amazon ES domain.

  2. The CloudWatch Logs destinations created to stream log events, have Kinesis Data Streams as their target.

  3. Once the log events stream to Kinesis Data Streams, the service invokes an AWS Lambda function to transform each log event to an Amazon ES document, which is then put in Kinesis Data Firehose.

  4. Kinesis Data Firehose indexes the documents on the Amazon ES domain.

  5. Kinesis Data Firehose logs errors in CloudWatch and delivers the records to Amazon Simple Storage Service (Amazon S3) for low-cost storage.

  6. You can monitor Kinesis Data Firehose as it sends custom CloudWatch Logs containing detailed monitoring data for each delivery stream.

Visualization

This solution provides data visualization and exploration support using Amazon ES and Kibana. An Amazon ES domain is created inside an Amazon VPC, preventing public access to the Kibana dashboard. Access to the Kibana dashboard is secured using a VPC security group and an AWS Identity and Access Management (IAM) role.


          Visualization using Kibana in the Centralized Logging solution

Figure 4: Visualization using Kibana in Centralized Logging

This solution optionally launches a Microsoft Windows Jumpbox Server that can be used to access the Amazon ES cluster and Kibana dashboard. An administrator account is configured to provide the permission to access the Kibana dashboard. This solution uses an Amazon Cognito user pool and an identity pool for authentication and authorization. For additional information about this solution’s security, refer to Security.