Collect, analyze, and display Amazon CloudWatch Logs in a single dashboard with the Centralized Logging solution - Centralized Logging

Collect, analyze, and display Amazon CloudWatch Logs in a single dashboard with the Centralized Logging solution

Publication date: November 2016 (last update: September 2021)

The Centralized Logging solution collects, analyzes, and displays Amazon CloudWatch Logs in a single dashboard. Amazon Web Services (AWS) services generate log data, such as audit logs for access, configuration changes, and billing events. In addition to AWS log data, web servers, applications, and operating systems all generate log files in various formats. Consolidating, managing, and analyzing these different log files is challenging to customers.

This solution contains a suite of infrastructure services that deploy a centralized logging solution. You can collect Amazon CloudWatch Logs from multiple accounts and AWS Regions. It uses Amazon OpenSearch Service (successor to Amazon Elasticsearch Service) and Kibana, an analytics and visualization platform that is integrated with Amazon OpenSearch Service, to create a unified view of all the log events. In combination with other AWS managed services, this solution provides customers with a turnkey environment to begin logging and analyzing your AWS environment and applications.

This solution also includes a demo AWS CloudFormation template that deploys sample logs, which you can use for testing purposes. We recommend deploying this optional template so that you can test how the solution works with sample logs generated by the demo resources.

The information in this guide assumes basic familiarity of web, application, and operating system log formats. Working knowledge of Amazon OpenSearch Service and Kibana for creating and customizing your own dashboards and visualizations, is recommended.

This implementation guide describes architectural considerations and configuration steps for deploying the Centralized Logging solution in the AWS Cloud. It includes links to CloudFormation templates that launch, configure, and run the AWS compute, network, storage, and other services required to deploy this solution on AWS, using AWS best practices for security and availability.

The guide is intended for IT infrastructure architects, administrators, and DevOps professionals who have practical experience architecting in the AWS Cloud.