REL06-BP07 Monitor end-to-end tracing of requests through your system
In a distributed system, a single request touches many components, and a failure or slowdown in any one of them affects the whole. Tracing follows each request across those components so you can pinpoint where errors and latency originate. This shortens root-cause discovery and reduces the mean time to resolve incidents.
Desired outcome:
-
You trace requests across every component, from client to compute to storage.
-
Your traces correlate with logs, metrics, and alarms so responders move from symptom to cause quickly.
-
Your tracing includes synthetic canaries and real-user monitoring to capture client-side interaction and latency.
-
You use a single, integrated tracing solution across cloud-native and third-party components.
Common anti-patterns:
-
Teams trace some components but not all, so latency from an untraced component stays hidden.
-
Teams omit synthetic canaries and real-user monitoring, which leaves client interaction out of the trace.
-
Teams run cloud-native and third-party tracing tools without integrating them into one solution.
-
Teams treat tracing as optional instrumentation and add it only after an incident exposes the gap.
Benefits of establishing this best practice:
-
You see the full picture of component interactions when responding to an issue.
-
You spend less time investigating because tracing makes root causes visible.
-
You make better and faster decisions, such as when to invoke failover or where to add self-healing.
-
You evaluate performance against service level agreements and objectives using client-side measurements.
Level of risk exposed if this best practice is not established: Medium
Implementation guidance
A correlation identifier travels with a request and ties together every operation on its behalf. With that identifier, a tracing tool collects the spans a request generates and assembles them into a service map that shows how components interact. A trace that covers the web tier but not the database leaves a blind spot where latency problems occur. Include every component in request traces: service clients, middleware gateways and event buses, compute, and storage such as key-value stores and databases.
When a component is uninstrumented, its latency does not disappear from the trace. It appears as time spent on neighboring components. A function with cold starts, for example, can make a downstream call appear slow when the delay is in the caller. Client-side behavior is the most commonly omitted segment. Add synthetic canaries and real-user monitoring to capture the interaction and latency remote clients experience. This is how you measure performance against customer-facing agreements.
When you run both cloud and third-party tracing tools, no single system holds all the traces, so you cannot correlate them into one view. Choose one primary tracing solution and integrate everything into it. If the primary solution is cloud-based, instrument components with their SDKs. If it is a third-party tool, configure it to ingest cloud-based trace telemetry.
AWS X-Ray (part of Amazon CloudWatch) collects application telemetry that you can visualize and filter across payloads, functions, traces, services, and APIs. You can turn on X-Ray for many components with no-code or low-code configuration. Amazon CloudWatch application monitoring combines traces with metrics, logs, and alarms in the X-Ray trace map. It also includes synthetics for endpoint monitoring and real-user monitoring for web clients. For open standards and mixed environments, use AWS Distro for OpenTelemetry to instrument applications and feed X-Ray.
Implementation steps
-
Turn on tracing for supported AWS services: Enable AWS X-Ray on native services such as Amazon S3, AWS Lambda, and Amazon API Gateway using configuration toggles in infrastructure as code, the AWS SDKs, or the console.
-
Instrument your application code: Instrument your components with AWS Distro for OpenTelemetry and X-Ray or a third-party collection agent. Follow the AWS X-Ray Developer Guide for language-specific instructions on tracing HTTP requests, SQL queries, and other processes.
-
Extend tracing to the client edge: Use X-Ray tracing with CloudWatch Synthetics canaries and CloudWatch RUM so you can analyze the request path from the end-user client through downstream infrastructure.
-
Correlate traces with alarms: Configure CloudWatch metrics and alarms from resource health and canary telemetry so teams are alerted quickly, then use the X-Ray trace map to move from an alarm into the related traces.
-
Integrate a third-party tracing tool if used: If a third-party tool is your primary solution, enable its X-Ray integration, such as Datadog
, New Relic , or Dynatrace , so all trace telemetry lands in one place.
Resources
Related best practices:
Related documents:
Related videos:
Related examples:
Related services: