Centralized Logging on AWS
Centralized Logging on AWS

Architecture Overview

Deploying this solution builds the following environment in the AWS Cloud.

        Centralized Logging Architecture

Figure 1: Centralized logging solution architecture on AWS

This solution includes an AWS CloudFormation template that you deploy in the primary account. This template launches an Amazon Elasticsearch Service (Amazon ES) domain, which is the hardware, software, and data exposed by Amazon Elasticsearch Service endpoints. During initial configuration of the solution’s primary template, users choose from one of three solution sizes to determine the number and type of data nodes (Amazon Elasticsearch instances) in the cluster: small, medium, or large.

The primary template also launches two Amazon EC2 instances in two separate Availability Zones of an Amazon Virtual Private Cloud (Amazon VPC) network. Two instances (one in each Availability Zone) are configured with an Nginx proxy to limit the exposure of data stored in Amazon ES. Each proxy server acts as an intermediary between the Kibana client web browser and the Amazon ES domain endpoint, filtering requests and then forwarding them to Amazon ES from a single, authenticated IP address (see Additional Security Settings for more information). During initial configuration, the user also specifies custom login credentials that offer an extra layer of protection. This highly available design uses an Application Load Balancer to distribute traffic to the proxy servers, and also enables automatic recovery to maintain instance availability.

The solution also includes a secondary template that you can deploy in secondary accounts or other AWS Regions. This template launches an AWS Lambda function that indexes logs from the secondary account or region on the Amazon ES domain in the primary account or region. During configuration of this template, you specify the Amazon ES domain endpoint and the Amazon Resource Name (ARN) of the primary AWS Identity and Access Management (IAM) role that the Lambda function will assume.

The centralized logging solution is designed to allow you to centralize the management of your own logs, but it also includes sample logs you can deploy for testing purposes. For more information, see Appendix A.