OPS02-BP04 Regularly analyze workload logs
Logs are the most detailed telemetry a workload produces, recording every request, error, and component decision. At production volume, no human being is reading logs line by line. Regular log analysis uses queries and machine assistance to extract patterns and answers from that detail.
Desired outcome:
-
You analyze your logs on a defined cadence, not only when an incident forces you to.
-
You know the frequent patterns in your logs, so a new error signature stands out on sight.
-
When you troubleshoot, you find the log events that explain a symptom quickly, wherever in the workload they were written.
-
You set each log group's retention and storage class to match the value you get from analyzing it.
Common anti-patterns:
-
Logs are collected diligently but only read after a critical issue has already surfaced.
-
Teams search logs manually and eyeball, leaving the query, pattern, and anomaly tools unused.
-
Every log group is retained indefinitely in full-price storage regardless of how it is used.
-
Analysis stops at account boundaries, so behavior that spans accounts goes unread.
Benefits of establishing this best practice:
-
Operational bottlenecks and security threats surface proactively, before they become incidents.
-
Machine assistance reads the volume a human cannot, so unusual events are seen rather than missed.
-
Debugging accelerates because the log detail behind a symptom is one query away.
-
Log spend stays proportionate to value instead of growing with volume by default.
Level of risk exposed if this best practice is not established: Medium
Implementation guidance
A metric shows that something changed, a trace shows where in the request path it happened, and the logs show the reason, such as the error text, the input that triggered the error, and the state the component recorded. Analyze logs on the same cadence and against the same baselines defined in OPS02-BP03 Regularly analyze workload metrics.
A workload can produce millions of log events an hour. Use queries and machine analysis instead of reading them manually. Pattern analysis condenses those millions of events into a short list of recurring shapes you can review. Comparing patterns between time ranges shows what changed after a deployment or at the start of an incident. Anomaly detection runs continuously between reviews, learning each log group's normal output and flagging what does not fit.
When a query matches more events than a person can read,
generate a natural language summary of the query results directly in
Log Analytics. Amazon CloudWatch investigations and AWS DevOps
Agent read log data when correlating a problem. AI agents can also
gather log data through the
Model Context Protocol servers for AWS
Some questions need more than one log group to answer. A request that touches several services leaves entries across several log groups, so correlate them by joining log groups on shared fields such as a request ID. CloudWatch Logs Insights supports joins across log groups and subqueries that filter one log group by what is found in another. Index the fields you join and filter on most to keep queries fast and reduce the volume scanned.
With Transaction Search enabled, spans are stored as structured logs in CloudWatch Logs and can be queried alongside application logs from the same request. Analyzing trace data itself is covered in OPS02-BP05 Regularly analyze workload traces.
OpenTelemetry treats logs as one of its three signals
Amazon's practice, described in
Instrumenting distributed systems for operational visibility
Match each log group's storage class and retention to how it is used: logs you alarm on need full capability, logs kept for occasional queries can use a lower-cost class, and logs nobody has queried in a year might not need their current retention.
Amazon CloudWatch Logs holds the log groups, runs the queries, learns the patterns, and connects log signals to the metrics and alarms in the rest of your observability practice. Log Analytics is the default console for this work. It combines queries, real-time streaming, and top-contributor analysis in one place, with saved parameterized queries, facets for exploring results, and query generation from natural language.
For workloads instrumented with OpenTelemetry, CloudWatch Omni queries the same logs with SQL, drafted from a plain language question if you prefer. From a trace's detail view, you can open the log records that carry that trace ID. This works only when each log record contains the trace ID, which can be injected using the previously described logging bridge.
Implementation steps
-
Bring logs where analysis happens: Confirm the workload's components send logs to CloudWatch Logs, and enable cross-account observability where the workload spans accounts, so one query can read the whole footprint. The collection and structure of these logs is developed in OPS01-BP01 Develop application telemetry.
-
Enable log anomaly detection: Create log anomaly detectors on the log groups that matter, so unusual patterns are flagged as they appear rather than waiting for the next review. When an anomaly points at a broader problem, you can start a CloudWatch investigation to correlate it with related telemetry and recent changes.
-
Query logs instead of reading them: Work in Log Analytics to extract answers from log data with CloudWatch Logs Insights queries, written for you from a natural-language prompt or manually. When a query's results are still more than a person can read, generate a natural language summary of them. Review pattern analysis to see recurring event shapes at a glance, and compare time ranges to isolate what changed after a deployment or at incident onset. For analysis beyond what queries express, enable the S3 Tables Integration so your logs are also available as managed Apache Iceberg tables, queryable in place by Amazon Athena, Amazon Redshift, or any Iceberg-compatible engine, and correlatable with data that lives outside CloudWatch.
-
Correlate across log groups: Answer request-path questions by joining log groups on shared fields such as request IDs, and use subqueries to filter one group by what you find in another. Create field indexes on the fields you join and filter on most, to speed queries and cut the volume they scan.
-
Watch live when it matters: Use Live Tail to stream log events in real time during deployments and incident response, when waiting for the next query cycle is too slow.
-
Turn recurring log signals into metrics: Create metric filters for the log events you keep querying for, so they become metrics you can chart and alarm on. Counting each distinct error reason separately keeps the resulting error metrics readable instead of noisy. Use Contributor Insights to rank the top contributors inside high-cardinality fields such as caller IPs or user agents.
-
Let assistants read at scale: When a flagged anomaly or a deviating metric needs explaining, start a CloudWatch investigation from it, so related telemetry, the logs included, is correlated into a hypothesis of the probable cause. Use AWS DevOps Agent to read operational signals, including log data, proactively between reviews, surfacing concerning trends with prioritized improvements. AI agents can also gather log data on request through the CloudWatch MCP server
, and whatever the source of a result set too large to read, generate a natural language summary of it. Route machine findings to the teams that own the affected components, and confirm them against the patterns and baselines this practice maintains. -
Match storage to analytical value: Assign the Infrequent Access log class to log groups you keep for occasional analysis rather than alerting, and set retention per log group based on how far back its analysis reaches. Revisit these choices as usage changes.
Resources
Related best practices:
Related documents:
Related videos:
Related workshops:
Related services: