View a markdown version of this page

FND05-BP04 Configure standardized locations for logs, traces, and metrics - AWS Well-Architected Framework

FND05-BP04 Configure standardized locations for logs, traces, and metrics

Security teams use logs and findings to analyze events that might indicate unauthorized activity or unintentional changes. Operations teams use logs, distributed traces, and metrics to understand system behavior and detect and mitigate issues early. Capture logs, traces, and metrics in standardized locations to correlate data points across all three signal types and simplify tool integrations.

Desired outcome:

  • You have a standardized approach to collect, analyze, and visualize log data, distributed traces, metrics, and security findings across workload accounts.

  • Security and operations teams can efficiently correlate, analyze, and visualize security data across disparate systems to discover potential issues and identify anomalies.

  • Security information and event management (SIEM) systems or other mechanisms are integrated to query and analyze log data for timely responses, tracking, and escalation of security events.

Common anti-patterns:

  • Teams independently own, manage, and collect logs and metrics in a way that is inconsistent with the organization's logging strategy.

  • Teams lack adequate access controls to restrict visibility and alteration of the data collected.

  • Teams don't govern their security logs, findings, and metrics as part of their data classification policy.

  • Teams neglect data sovereignty and localization requirements when configuring data collections.

Benefits of establishing this best practice:

  • Standardized collection and querying of log data improves the quality and speed of insights derived from analysis.

  • Automated lifecycle management for log data reduces storage costs and maintains compliance with retention requirements.

  • Fine-grained access controls protect sensitive log information and confirm that teams have appropriate access based on their operational needs.

  • Centralized logging supports integration with security and analytics tools to correlate events, visualize trends, and accelerate incident response.

  • Unified observability across logs, traces, and metrics supports faster root cause analysis by correlating multiple data types in a single location.

Level of risk exposed if this best practice is not established: Medium

Implementation guidance

Collecting activity data from distributed workloads for centralized analysis is challenging. Security teams use SIEM systems to collect data from these sources and perform correlation, analysis, and response workflows, but managing permissions for disparate data sources and operating ETL processes introduces overhead.

Aggregate all relevant sources of security log data into a dedicated Log Archive account, as described in Organizing Your AWS Environment Using Multiple Accounts. Include workload security data and AWS service logs (AWS CloudTrail, AWS WAF, Elastic Load Balancing, and Amazon RouteĀ 53). Capturing data in a separate account with cross-account permissions helps prevent log tampering by users with broad access in the workload account, provides a single integration point for additional tools, and simplifies data retention and lifecycle management. Evaluate data sovereignty, compliance scope, and other regulations to determine whether multiple storage locations and retention periods are required.

Amazon CloudWatch Unified Data Store is the standardized analytics-ready location for your security logs and findings. It consolidates operational, security, and compliance data in one place across multiple AWS accounts and Regions, supports Open Cybersecurity Schema Framework (OCSF) and OpenTelemetry formats, and is accessible from CloudWatch Logs Insights, Amazon Athena, Amazon OpenSearch Service, Amazon SageMaker AI, and third-party Iceberg-compatible tools without data duplication. Use cross-account cross-Region log centralization to copy log data from your organization into a central account and Region, and aggregate common security sources such as CloudTrail, RouteĀ 53 Resolver query logs, VPC Flow Logs, compute and workload logs from Amazon EC2, Amazon ECS, Amazon EKS, and AWS Lambda, and findings from AWS Security Hub CSPM, Amazon GuardDuty, and Amazon Inspector into the same store. Pair the unified data store with Amazon S3 for long-term retention of raw logs and low-cost archival.

Deploy security analytics tools that operate in AWS into a Security Tooling account separate from your Log Archive account. This separation implements defense in depth, protecting the integrity and availability of logs and log management independently from the tools that access them. Services like Amazon Athena run on-demand queries across multiple data sources, AWS Quick provides visualization, and AI-powered solutions can translate findings into human-readable summaries and support natural language interaction.

Apply the same centralization and standardization principles to distributed traces and metrics. Application Observability (APM) and AWS Distro for OpenTelemetry capture end-to-end request flows across distributed systems, so you can understand request dependencies and pinpoint latency or error sources across microservices. Correlate traces with the CloudTrail and application logs in your Log Archive account so investigators can move from a suspicious event to the exact user requests that triggered it.

Metrics provide counts, rates, and durations that make anomalies measurable. Amazon CloudWatch is the central metrics repository for AWS services and custom application metrics, and CloudWatch cross-account observability aggregates metrics, logs, and traces from workload accounts into a monitoring account for unified dashboards and alarms. For long-term retention and downstream analytics, use CloudWatch metric streams to deliver metrics into Amazon S3 alongside your other observability data, so you can query and correlate logs, traces, and metrics together.

Implementation steps

  1. Create the Log Archive and Security Tooling accounts: Using AWS Organizations, create the Log Archive and Security Tooling accounts under a security organizational unit. If using AWS Control Tower, these accounts are created automatically. See AWS Control Tower account factory. Follow AWS SRA Logging Architecture guidance.

  2. Evaluate data sovereignty and compliance requirements: Determine whether your organization requires multiple log storage locations based on data residency laws, regulatory scopes, or data classification policies. Workloads spanning multiple geographic regions with strict data localization requirements might need per-Region Log Archive accounts or Region-specific storage locations in a single account. Document retention requirements for different log types and sensitivity levels so they inform your storage and lifecycle configuration in later steps.

  3. Configure your standardized security data locations: Use Amazon CloudWatch Unified Data Store as the destination for operational, security, and compliance telemetry, and cross-account cross-Region log centralization to aggregate log data across your organization. Capture security data from AWS Regions that are opted-in for your accounts, even when not actively in use. Pair the unified data store with raw log storage in Amazon S3 for long-term retention and integration with third-party data products where required.

  4. Configure data retention requirements: Establish appropriate lifecycles for each log source using Amazon S3 Lifecycle and Log retention in CloudWatch Logs. Typically, three to twelve months of logs are readily available, with retention up to seven years. Align these settings with your security requirements and regulatory mandates.

  5. Configure data source publication to your standardized locations: Identify sources and configure them to publish to your standardized locations. Evaluate options that automatically export data in the desired format rather than requiring ETL. Use cross-account cross-Region log centralization to replicate log groups from member accounts into your CloudWatch Unified Data Store, and CloudWatch metric streams to deliver metrics into the same data store. Include compute and workload logs from Amazon EC2, Amazon ECS, Amazon EKS, and AWS Lambda so investigators can correlate API activity with what happened inside the workloads. Configure findings from AWS Security Hub CSPM, GuardDuty, and Amazon Inspector to flow into CloudWatch alongside raw logs, and publish third-party sources via their OCSF- or OpenTelemetry-compatible exporters.

  6. Configure tools to access your standardized locations: Configure tools like Amazon Athena, AWS Quick, Amazon OpenSearch Service, or third-party solutions with required access. Operate these tools out of the Security Tooling account with cross-account read access to the Log Archive account. Analysts can query the same OCSF- and OpenTelemetry-formatted data in the CloudWatch Unified Data Store from CloudWatch Logs Insights, Athena, OpenSearch Service, and third-party Iceberg-compatible platforms without duplicating data.

  7. Centralize traces and metrics alongside logs: Turn on AWS X-Ray for application tracing, or deploy the AWS Distro for OpenTelemetry Collector for standardized telemetry that works across AWS and third-party backends. Configure CloudWatch cross-account observability so metrics, logs, and traces from workload accounts are visible in a central monitoring account. Use CloudWatch metric streams to deliver metrics into Amazon S3 for long-term retention and correlation with logs.

  8. Use logs for alerting: Configure alerting using Amazon GuardDuty, AWS Security Hub CSPM, and AWS Config. Enable AWS Security Hub CSPM alongside Security Hub CSPM. AWS recommends enabling both services so Security Hub CSPM findings are routed to Security Hub CSPM automatically and correlated with findings from Amazon Inspector, Amazon Macie, and GuardDuty to generate exposure findings. Use custom alert generation engines for specific alerts relevant to your environment, and Amazon EventBridge for custom event-driven alerting.

Resources

Related best practices:

Related documents:

Related examples:

Related tools:

Related services: