PERF05-BP02 Evaluate your workload performance
Everyday performance does not predict how a workload behaves at peak traffic. Test the workload against your KPIs under realistic and stressful conditions to find where it slows or stops scaling before customers notice.
Desired outcome:
-
You test the whole workload against defined KPIs, not individual components in isolation.
-
Your test environment reproduces the production factors that affect performance, including configuration, scale, data distribution, quotas, and dependencies.
-
You test at normal load, peak load, and beyond expected load to find limits and time-dependent degradation early.
-
Your performance tests run automatically as part of the delivery pipeline and compare results against thresholds and previous baselines.
-
For AI workloads, your pass criteria cover response quality or task success together with latency, throughput, and resource efficiency.
-
You adjust resource configurations using test evidence instead of guesswork.
Common anti-patterns:
-
Teams run load tests without defining the KPIs the test is meant to validate.
-
Teams test individual parts of the workload but never exercise the whole system together.
-
Teams test on infrastructure that does not reproduce the production factors affecting performance, so the results do not transfer.
-
Teams test only to expected load and never beyond it, so they never see the breaking point.
-
Teams test the workload once and do not repeat the test as the environment changes.
-
Teams do not distinguish legitimate network stress tests from prohibited volumetric DDoS simulations or review the applicable provider testing policy.
-
Teams test AI workloads with a fixed prompt size and score only latency, missing token-distribution, throughput, and quality regressions.
Benefits of establishing this best practice:
-
You discover scaling limits and bottlenecks before they affect customers.
-
You prepare realistic test data and reuse it to make results comparable over time.
-
You anticipate the changes a growing workload will need instead of reacting to a disruption.
-
You right-size resources from evidence and avoid both over- and under-provisioning.
-
You gain confidence that each release still meets its performance requirements.
Level of risk exposed if this best practice is not established: Medium
Implementation guidance
To trust load-test results, exercise the complete workload. Testing components in isolation does not prove that shared dependencies handle combined load. Reproduce the production factors that affect performance: compute characteristics, scaling configuration, quotas, network paths, data volume, and dependencies. Match production closely enough that differences do not invalidate conclusions about bottlenecks or capacity. Every test also needs KPIs. Without targets, results cannot show whether the workload meets its performance requirement.
Use more than one test type, because each reveals a different failure mode. Load tests validate expected and peak traffic. Stress tests push beyond expected capacity to identify saturation and the breaking point. Spike tests apply a sudden traffic surge to check burst scaling and recovery. Endurance tests hold load over time to expose memory leaks, connection-pool exhaustion, queue growth, and other time-dependent degradation. These tests show whether the workload degrades gracefully, fails abruptly, or scales non-linearly.
Run tests on synthetic or sanitized data, with sensitive and identifying fields removed, to protect customers and keep results repeatable. Data volume and distribution also shape performance: a test against a few thousand uniform records does not predict behavior on a production-sized, skewed data set. For generative AI and agentic workloads, cover the expected distribution of input and output tokens, prompt categories, retrieval sizes, tool calls, model configurations, and concurrent sessions. Evaluate response quality or task success alongside time to first token, output token throughput, end-to-end latency, throttling, errors, and cost or resource consumption per task.
Every deployment, dependency upgrade, and traffic-pattern shift changes the performance profile. Include evaluation in the delivery pipeline so it runs on a schedule and compares each result against previous baselines. With cloud infrastructure, you can create a representative test environment and remove it afterward. Distributed Load Testing on AWS runs distributed test workers on Amazon ECS and AWS Fargate. It supports JMeter, k6, Locust, and simple HTTP tests, distributes load across Regions, and records baseline comparisons. Its optional MCP server supports AI-assisted result analysis, and its native AWS DevOps Agent integration investigates completed test runs for bottlenecks and root-cause hypotheses.
Review the
Amazon EC2 testing policy
Implementation steps
-
Prepare representative test data: Produce synthetic or sanitized data at production-representative volume and distribution, and remove sensitive and identifying information before the data leaves its source system. For AI workloads, include representative token lengths, prompt categories, retrieval inputs, tool paths, and expected output-quality cases.
-
Define objectives and KPIs: State which performance aspects the test evaluates and attach a target to each. Include latency percentiles, throughput, error rate, saturation point, and resource efficiency. For generative AI or agentic workloads, add time to first token, output token throughput, throttling, end-to-end task time, and response quality or task-success criteria. These targets are the pass or fail conditions the pipeline checks.
-
Build a representative environment: Reproduce the production characteristics that affect results, including compute types, autoscaling, service quotas, network paths, configuration, data size, and dependencies. Document every material difference from production and how it affects interpretation. Use on-demand infrastructure to create the environment for the test and remove it afterward.
-
Configure tooling and monitoring: Use a load testing tool matched to the workload. Distributed Load Testing on AWS supports simple HTTP, JMeter, k6, and Locust tests with distributed Fargate workers, multi-Region traffic, scheduling, and baseline comparison. Instrument the workload and the test generators with Amazon CloudWatch
so a load-generator limit is not mistaken for a workload limit. -
Run scenarios across the performance envelope: Run load tests at normal and expected peak traffic, stress tests beyond expected capacity, spike tests for sudden surges, and endurance tests for sustained behavior. Review the Amazon EC2 testing policy
for high-volume tests originating from EC2, and do not use a load test to perform a prohibited volumetric DDoS simulation. -
Analyze results, validate findings, and adjust resources: Compare p50, p95, and p99 latency, throughput, error rate, saturation, and resource use against KPIs and previous baselines. Use DLT baseline comparison and, where configured, AWS DevOps Agent to accelerate investigation of regressions. Treat agent findings as hypotheses, verify them against telemetry, change one factor at a time, and rerun the test before accepting a remediation.
-
Document, share, and iterate: Record findings and recommendations and share them with stakeholders to support optimization decisions. Repeat the evaluation on a regular cadence and after any significant change to the workload.
Resources
Related best practices:
Related documents:
Related videos:
Related examples:
Related services: