View a markdown version of this page

Evaluate agent quality - Amazon CloudWatch

Evaluate agent quality

Agents fail in ways that do not show up as errors: the run completes, but the answer is wrong, incomplete, or against your policies. Evaluation is how you measure response quality, so you can catch these failures and tell whether a change to a prompt, a model, or a tool made your agent better or worse.

How evaluation fits together

The evaluation workflow builds from your real traffic:

  1. Curate test cases from real traces. Your traces already contain real requests and the behavior that needs to improve. You collect representative runs (common requests, edge cases, known failures) into a dataset, a fixed test set you can run against repeatedly. See Datasets.

  2. Score responses with evaluators. An evaluator reads a trace (the input, the response, and the steps in between) and returns a score against a criterion such as correctness or relevance. The scores from running one or more evaluators are an evaluation, the record you compare against later runs. You can score a single trace as you debug, or score your agent's outputs across a whole dataset. See Evaluators and evaluations.

  3. Compare versions with experiments. An experiment runs two or more agent versions against the same dataset with the same evaluators, so a score difference can only come from the change you made. See Experiments.

The playground sits alongside this loop for interactive iteration: try a prompt or an agent change and see the effect immediately, then formalize what worked into a dataset and an experiment. See Prompt playground.

The agent evaluation workflow: a development loop of curate, score, and compare, with the playground alongside for interactive iteration, and an online evaluation in production scoring a sample of live traffic whose failing traces feed back into the loop as new test cases.

Where evaluators come from

Every evaluator runs through Amazon Bedrock AgentCore Evaluations, so one catalog covers three kinds:

  • Built-in — ready-made checks for common criteria such as correctness, helpfulness, and tool-use quality. Start here.

  • Third-party — metrics from the DeepEval and AutoEval open-source libraries, run for you, so you can use familiar metrics without hosting them.

  • Custom — evaluators you write yourself, including one that reuses a built-in or third-party evaluator's scoring on a judge model you choose.

Choose an evaluator by the criterion you want to check; you can mix kinds in the same evaluation. For the full catalog and more on how to write your own evaluators, see Evaluators and evaluations. To understand how Amazon Bedrock AgentCore evaluates agent performance, see Evaluate agent performance and Third-party evaluators in the Amazon Bedrock AgentCore Developer Guide.

Evaluation during development and in production

The same pieces serve two workflows. During development, you run the loop deliberately: build a dataset, score a change in the playground, confirm it with an experiment before you deploy. In production, evaluation runs unattended: an online evaluation is a configuration that continuously scores a sample of your live traffic with the evaluators you choose (see Online evaluations), and a declining pass rate shows that quality regressed. When production evaluation catches a regression, the development loop is how you fix it. See Tutorial: Fix a production quality regression for the end-to-end workflow.

Evaluators read trace content, so your agent must send traces that include input and output. See Send AI agent telemetry. For the trace, span, and session vocabulary these pages build on, see Monitor AI agents. To monitor quality signals across your agents in production, see Monitor agents in the web UI.