View a markdown version of this page

OPS02-BP05 Regularly analyze workload traces - AWS Well-Architected Framework

OPS02-BP05 Regularly analyze workload traces

A trace shows the path a request takes through your workload. Trace analysis looks at these paths at the scale of your entire architecture. While metrics quantify changes and logs record what each component did, traces show how the parts of your workload behaved together: which service added latency, which dependency returned an error, and which interaction pattern is drifting.

Desired outcome:

  • You analyze your trace data on a defined cadence, not only when an incident forces you to.

  • You find slow and failing request paths in your trace data before users report them.

  • Your trace analysis reads the RED metrics (rate, errors, duration) derived from traces, not only individual timelines.

  • Your traces connect to the logs and client-side telemetry behind them, so one analysis explains a request from frontend to backend.

Common anti-patterns:

  • Traces are collected but overlooked, and analysis relies on logs and metrics alone.

  • Trace data is not correlated with the logs recorded during the same request.

  • The RED metrics derived from traces go unread per service and dependency.

  • Analysis only ever inspects single traces, so patterns across thousands of requests stay invisible.

Benefits of establishing this best practice:

  • Troubleshooting improves and mean time to resolution falls, because the failing service is found rather than guessed.

  • Dependencies and their impact on the request path become visible and quantified.

  • Performance issues are identified and corrected swiftly, including the ones only some requests experience.

  • Trace-derived metrics inform decisions about where optimization effort pays back.

Level of risk exposed if this best practice is not established: Medium

Implementation guidance

This practice assumes traces exist (for detail on instrumenting and collecting traces, see OPS01-BP04 Develop distributed tracing). The cadence and baseline discipline come from OPS02-BP03 Regularly analyze workload metrics. Trace analysis joins that same review and does not need its own schedule. Traces answer questions metrics and logs cannot: where in the request path the workload is slow, for which requests, and why.

Trace analysis needs both a close view of single requests and a wide view across many. The wide view starts with the RED metrics derived from traces: request rate, errors, and duration per service and dependency. Drift in those aggregates tells you which service's traces are worth opening. From there, work from the service to its slowest operations, from the operations to the requests in the tail, and from a tail request to the exact call that put it there.

Store spans as structured data so you can query them precisely. Which checkout requests over five seconds touched the payment service yesterday? Which customers hit the failing dependency? Does the latency follow a specific attribute your instrumentation emits? Query every transaction to measure who is affected and by how much. Use OpenTelemetry's trace semantic conventions as the attribute vocabulary (as described in OPS01-BP01 Develop application telemetry). The same query works across services regardless of which SDK produced the span, because attribute names are standardized.

Include client-side traces from real users and synthetic canaries alongside backend traces, so client-experienced latency ties to the backend call that caused it. Correlate traces with logs: the trace locates the failing service, and the logs recorded during that request explain it. Extend the analysis view across every account the workload runs in, so one investigation follows a request from the frontend to the log line.

Enable Amazon CloudWatch Application Signals as the trace-analysis entry point. You get per-service views of the RED metrics derived from trace data, dependency breakdowns, and the correlated traces for each. The AWS X-Ray console remains the trace backend's own analysis surface, and its insights feature raises anomalies automatically. AI assistants extend the same reading: CloudWatch investigations correlates trace data with related telemetry when something needs explaining, AWS DevOps Agent reviews it proactively, and MCP servers make trace data available to AI agents from wherever they work.

When a traced workload is an AI agent, a trace that completed quickly and without errors can still contain a well-formed response that is wrong. In this scenario, CloudWatch Omni scores agent responses with your chosen evaluators on a sample of production traffic or on demand against a dataset, and then shows the scores on the trace next to its latency, errors, and token usage. You can read these scores as one more trace-derived signal in your workload review. For more detail on evaluation as a discipline, see AGENTOPS06 Testing, evaluation, and validation frameworks.

Implementation steps

  1. Start from the service views: With Application Signals enabled, begin each review on the Services page: compare each service's RED metrics and dependency latencies against your baseline, and open the service detail to reach the traces behind a drifting number. The trace-derived aggregates decide where deeper trace reading is worth the time.

  2. Set objectives on the RED metrics: Create service level objectives on the services and operations that matter most, using the same RED metrics the service views read: Application Signals offers the duration and errors it already collects, as latency and availability, for ready-made service level indicators. It tracks attainment against the goal you set and can alarm on error-budget burn rate, so a fast burn interrupts you instead of waiting for the next review. Those alarms, for an objective at risk or already breached, can be configured to create investigations automatically, so the correlation work starts before you arrive. The SLO dashboard answers each review's first question, whether the workload is meeting its objectives, at a glance.

  3. Query spans across every transaction: With Transaction Search enabled, spans are stored in OpenTelemetry format in CloudWatch Logs, so you can search and analyze them with the attribute names your instrumentation emits. Quantify the tail: how many requests, which users, which shared attribute. In CloudWatch Omni, the same spans are queryable with SQL from Explore, including a lookup of one trace ID across every application and agent that handled it.

  4. Let anomalies find you between reviews: Enable X-Ray insights so anomalies in trace data, such as rising fault rates or latency, are raised automatically with their probable root cause and a timeline. Review open insights as part of the cadence, and use groups to scope insights to the request paths that matter most.

  5. Explore patterns interactively: Use X-Ray Analytics to compare trace sets against each other, such as the slow tail against the healthy baseline, and identify the attributes that distinguish them. Filter into the latency distribution rather than averaging over it.

  6. Correlate traces with logs: Enable trace to log correlation so the log events recorded during a traced request are reachable from the trace itself. The trace locates the failing service. The correlated logs explain it.

  7. Include the client's view: Analyze traces that begin at real users and synthetic canaries alongside backend traces, so client-experienced latency is attributed to the backend call that caused it, and regressions are caught by canaries before users meet them. Developing this telemetry is covered in OPS01-BP02 Develop user experience telemetry.

  8. Let assistants read at scale: When an SLO metric or a flagged anomaly needs explaining, start a CloudWatch investigation from it, so trace data is correlated with related telemetry and recent changes into a hypothesis of the probable cause. Use AWS DevOps Agent to review operational signals, trace data included, proactively between reviews. AI agents can also analyze the same data from wherever they work through the Application Signals MCP server, which reads service health, SLO compliance, and the spans behind them. Route machine findings to the teams that own the affected components, and confirm them against the baselines this practice maintains.

  9. Span the whole footprint: Configure Transaction Search across accounts with CloudWatch cross-account observability, so trace analysis follows requests wherever the workload runs instead of stopping at an account boundary.

Resources

Related best practices:

Related documents:

Related videos:

Related workshops:

Related services: