Overview
Amazon Web Services (AWS) provides service-specific operational metrics and log files to give customers insight into how the service is operating. Many AWS services also generate security log data, including audit logs for access, configuration changes, and billing events. In addition to AWS log data, web servers, applications, and operating systems generate log files in different formats, and in a disorganized and distributed fashion. Effectively consolidating, managing, and analyzing these different log types is a challenge for almost every company, which is why many AWS customers choose to implement a centralized logging solution.
The AWS Cloud provides a suite of infrastructure services that enable you to deploy a centralized logging solution in an available and affordable way. This guide provides infrastructure and configuration information for deploying a centralized logging solution that collects, analyzes, and displays logs on AWS across multiple accounts and AWS Regions. The solution uses Amazon Elasticsearch Service (Amazon ES), a managed service that simplifies the deployment, operation, and scaling of Elasticsearch clusters in the AWS Cloud, as well as Kibana, an analytics and visualization platform that is integrated with Amazon ES. In combination with other AWS managed services, this solution provides customers with a turnkey environment to begin logging and analyzing their AWS environment and applications.
The information in this guide assumes basic knowledge of web, application, and operating system log formats. It is also helpful to have working knowledge of Amazon ES and Kibana for creating and customizing your own dashboards and visualizations.
Cost
You are responsible for the cost of the AWS services used while running this reference deployment. As of the date of publication, the cost for running a centralized logging solution with this solution’s default settings in the US East (N. Virginia) Region is as shown in the table below. This includes charges for Amazon Elasticsearch instance hours and storage, Amazon EC2 usage, and base pricing for an Application Load Balancer.
| Cluster Size | Total Cost/Hour |
|---|---|
| Small | $0.53 |
| Medium | $3.31 |
| Large | $16.90 |
This cost estimate does not reflect variable, usage-driven charges incurred from Amazon CloudWatch, AWS CloudTrail, and AWS Lambda, and Application Load Balancer data throughput, or the cost for sample logs. For full details, see the pricing webpage for each AWS service you will be using in this solution.
