Centralized Logging on AWS
Centralized Logging on AWS

Design Considerations

Custom Sizing

Choose from three preset Amazon ES cluster sizes to support your anticipated log traffic:

  • Small:

    • 3 dedicated master nodes; c4.large.elasticsearch instance type

    • 4 data nodes; i3.large.elasticsearch instance type

  • Medium:

    • 3 dedicated master nodes; c4.large.elasticsearch instance type

    • 6 data nodes; i3.2xlarge.elasticsearch instance type

  • Large:

    • 3 dedicated master nodes; c4.large.elasticsearch instance type

    • 6 data nodes; i3.4xlarge.elasticsearch instance type

Scalability

Modify your cluster’s instance count and type directly in Amazon ES to accommodate your changing environment and requirements, without having to reconfigure the solution architecture or manage backend resources. As a best practice, we recommend that you monitor your cluster’s performance metrics.

Kibana Dashboard

Take advantage of Kibana features to create, save, and share custom visualizations and customer views. This solution includes a configuration file to get you started with some popular dashboard views.

Logging Across Accounts and Regions

The Amazon ES domain that this solution creates can accept log data from other AWS accounts and AWS Regions. Customers can launch the spoke template in secondary accounts and other regions to use this solution to index logs across accounts and regions.

During initial configuration, enter the secondary account IDs in the Spoke Accounts parameter before you deploy the spoke template in those accounts to ensure that the secondary accounts can assume the master IAM role. To add accounts after you launch the primary template, update the Spoke Accounts parameter in the primary stack with the secondary account IDs. Then, update the primary stack and deploy the spoke template in the secondary accounts. You can remove an account at any time by removing its ID from the Spoke Accounts parameter.

Regional Deployments

This solution uses Amazon Cognito which is available in specific AWS Regions only. Therefore, you must launch this solution’s primary template in a region that supports Amazon Cognito. (For the most current service availability by region, see AWS service offerings by region.) The solution’s spoke template can be deployed in any region in secondary accounts. Once deployed, the solution will monitor logs for all regions in applicable accounts.