Centralized Logging on AWS
Centralized Logging on AWS

Design Considerations

High Availability

Amazon Elasticsearch Service is a managed service that automatically replicates and distributes your multi-node Elasticsearch cluster across different Availability Zones. This solution implements a multi-AZ architecture and uses an Application Load Balancer and automatic recovery to maintain availability of the proxy servers, which manage all client requests to the Amazon ES endpoint.

Custom Sizing

Choose from three preset Amazon ES cluster sizes to support your anticipated log traffic:

  • Small:

    • 3 dedicated master nodes; t2.small.elasticsearch instance type

    • 2 data nodes; r4.large.elasticsearch instance type

  • Medium:

    • 3 dedicated master nodes; r4.large.elasticsearch instance type

    • 4 data nodes; r4.2xlarge.elasticsearch instance type

  • Large:

    • 3 dedicated master nodes; r4.xlarge.elasticsearch instance type

    • 10 data nodes; r4.4xlarge.elasticsearch instance type

Scalability

Modify your cluster’s instance count and type directly in Amazon ES to accommodate your changing environment and requirements, without having to reconfigure the solution architecture or manage backend resources. As a best practice, we recommend that you monitor your cluster’s performance metrics.

Custom Reporting

Take advantage of Kibana features to create, save, and share custom visualizations and customer views. This solution includes a configuration file to get you started with some popular dashboard views.

Logging Across Accounts and Regions

The Amazon ES domain that this solution creates can accept log data from other AWS accounts and AWS Regions. Customers can launch the spoke template in secondary accounts and other regions to use this solution to index logs across accounts and regions.

During initial configuration, enter the secondary account IDs in the Spoke Accounts parameter before you deploy the spoke template in those accounts to ensure that the secondary accounts can assume the master IAM role. To add accounts after you launch the primary template, update the Spoke Accounts parameter in the primary stack with the secondary account IDs. Then, update the primary stack and deploy the spoke template in those accounts. You can remove an account at any time by removing its ID from the Spoke Accounts parameter.