PERF05-BP01 Use monitoring solutions to understand the areas where performance is most critical
Workload components contribute unequally to the customer experience, user-perceived latency, and operating cost. Instrument the workload to reveal those pressure points, and focus optimization on them.
Desired outcome:
-
You have instrumentation across the performance-relevant paths of the workload, from client sessions through application code to data stores.
-
Your metrics, traces, and logs are correlated so a single slow request is traceable across component boundaries.
-
You know which components sit on the critical path for customer-facing latency and cost.
-
Your dashboards show performance against business KPIs, not just raw resource counters.
-
Your team reviews performance data continuously, not only when an incident occurs.
Common anti-patterns:
-
Teams rely only on the default metrics emitted by their monitoring tool and never add workload-specific instrumentation.
-
Teams review metrics only when something breaks, so they have no baseline to compare against.
-
Developers instrument code without reference to the KPIs the workload is measured on.
-
Teams collect metrics but build no dashboards that relate them to business outcomes.
-
Teams select monitoring tools that are expensive to run and hard to maintain.
-
Teams treat each component in isolation and never correlate signals across layers to explain a system-wide slowdown.
-
Teams treat AI-generated investigation hypotheses as confirmed root causes without validating them against telemetry or subsequent tests.
Benefits of establishing this best practice:
-
You spend optimization effort where it changes the customer experience, not on components that are already fast enough.
-
You detect bottlenecks during traffic growth, before users report degradation.
-
You explain the root cause of a slowdown faster because signals from different layers line up.
-
You justify performance work with data that ties back to business KPIs.
-
You keep monitoring cost proportional to the value it delivers by instrumenting deliberately.
Level of risk exposed if this best practice is not established: High
Implementation guidance
Monitor to find the components with the greatest effect before your customers do. Instrumentation must reveal the critical path: the sequence of calls a user request travels, and the point on that path where time accumulates. Default metrics like CPU and memory show resource activity. They do not indicate whether a resource is on the critical path between a customer and the response they are waiting for.
Observability has three signal types. Metrics tell you that something is slow. Traces tell you where the time went across services. Logs tell you why a request behaved as it did. Metrics without traces surface a spike without its downstream cause. Traces without logs surface a slow database call without the query behind it. A shared request identifier links the spike, the span, and the log line.
Sampling, aggregation, and retention balance detail against cost. Decide up front which questions the data must answer.
Normal traffic does not reveal every bottleneck. Load testing and synthetic monitoring generate traffic on demand, so scaling limits and slow paths surface under controlled conditions before a real peak. Synthetic canaries also produce a consistent baseline for comparison with real user behavior.
Once you see the critical path, choose the statistics you track on it. Averages and P99 miss the small number of extreme outliers that take much longer than the rest. Some operations depend on a predictable completion time, such as a clone, backup, or provisioning step. For those, monitor the maximum or P100 alongside average and P99. Define an evaluation window and minimum sample count so a single outlier does not create a misleading alarm. Investigate extreme values separately from improvements to typical latency.
AWS provides monitoring capabilities for each performance-relevant
layer of a workload.
Amazon CloudWatch RUM captures performance from real client-side
sessions.
Amazon CloudWatch Application Signals provides application
performance monitoring, service maps, correlated telemetry, and
service level objectives. Instrument applications with
OpenTelemetry
For generative AI and agentic workloads, collect model, retrieval,
memory, and tool-call spans alongside infrastructure metrics.
Track signals such as time to first token, output token
throughput, quota pressure, end-to-end completion time, and task
success.
Amazon CloudWatch generative AI observability provides model and
agent views with prompt tracing, latency, token usage, and error
telemetry.
CloudWatch investigations and
AWS DevOps Agent correlate telemetry, code, and deployment
changes to surface bottleneck and root-cause hypotheses. Validate
agent-generated findings against the underlying telemetry and
confirm improvements with subsequent measurements or tests.
Amazon Managed Service for Prometheus
Implementation steps
-
Map the critical performance paths: Trace how a customer request flows through the workload and identify the components on that path that most affect latency, cost, or customer experience. Rank these areas so instrumentation and later optimization focus on the parts that matter most.
-
Instrument each performance-relevant layer of the workload: Use Amazon CloudWatch RUM for client sessions, Amazon CloudWatch Application Signals with OpenTelemetry or AWS Distro for OpenTelemetry
for application services and traces, and Amazon CloudWatch Database Insights with RDS Enhanced Monitoring for data stores. For generative AI and agentic paths, include model, retrieval, memory, and tool calls and collect latency, token, quota, error, and task-success signals. -
Generate traffic to expose bottlenecks: Set up Amazon CloudWatch Synthetics canaries to replay browser activity on a schedule and produce a consistent baseline. Use Distributed Load Testing on AWS to drive peak and growth-rate traffic and find where the workload stops scaling linearly.
-
Correlate, visualize, and investigate the signals: Bring metrics, traces, logs, and deployment events together so a single request is traceable across layers. Use Application Signals or build dashboards in Amazon Managed Grafana
to show performance against business KPIs. Use CloudWatch investigations or AWS DevOps Agent to accelerate analysis of unfamiliar bottlenecks, and validate their hypotheses against the source telemetry. -
Prioritize improvements and measure them: Review the instrumented critical areas with your team, agree on the highest-impact fixes, and apply changes one at a time. Measure each change against the baseline so you keep only the improvements the data confirms.
-
Target tail and worst-case latency on predictability-sensitive operations: Monitor P99 together with the maximum or P100 for operations where customers depend on a consistent completion time. Choose an evaluation window and minimum sample count based on request volume and the review cadence. Use CloudWatch percentile statistics for P100 and the Maximum statistic where each is supported, and treat extreme outliers as a distinct investigation from improving average or P99 latency.
Resources
Related best practices:
Related documents:
Related videos:
-
AWS re:Invent 2025 - Troubleshooting database performance issues at scale
-
AWS re:Invent 2025 - Implementing observability at scale: A blueprint for success
-
AWS re:Invent 2025 - Elevate application and generative AI observability
-
AWS re:Invent 2025 - AI-Native Era of Observability: How You Can Get Started Today
Related examples:
Related services: