Online evaluations

On-demand evaluation scores a trace when you ask. An online evaluation scores your traffic continuously: you choose the agents to watch, the evaluators to run, and how much traffic to sample, and CloudWatch Omni evaluates matching traces as they arrive. Quality regressions surface in your evaluation trends without anyone running a check. Online evaluation is the continuous-monitoring stage of the evaluation workflow described in Evaluate agent quality.

How an online evaluation works

An online evaluation is a configuration that you create, and then enable:

• Agents to evaluate — chosen by service name from the agents seen in your space's telemetry.

• Evaluators to run — one or more evaluators from the catalog described below.

• Sampling percentage — how much of the matching traffic is evaluated. Sampling is your cost control: judge-based evaluators invoke a judge model for every sampled trace, so evaluate a fraction of high-volume traffic.

• Session settings (optional) — a session timeout that determines when a conversation is considered complete, so session-level evaluators can score the full exchange. The timeout is measured from a session's last activity: a session is evaluated only after it has been idle for this long. A session that never goes idle — for example a long-running or continuously reused session that always has recent activity — is never treated as complete, so it is never evaluated and produces no results. Set the timeout shorter than the typical gap between turns in your sessions so they finalize and are scored.

While the configuration is enabled, CloudWatch Omni evaluates sampled traces continuously. Scores appear alongside the traces and feed the trends in the Evaluation dashboard. Each configuration also gets a dedicated CloudWatch log group holding its results in JSON, and publishes each score as a CloudWatch metric, so you can query results with CloudWatch Logs Insights and alarm on a score the same way you alarm on any other metric. Results follow the OpenTelemetry semantic conventions for generative AI evaluation events, and each result references the trace and session it came from, so you can go from a score back to the run that produced it.

To change an enabled configuration's evaluators, disable it first. The evaluator list is locked while the evaluation runs. A running evaluation also locks the custom evaluators it uses, so that scores stay comparable over time: you cannot edit or delete one while an evaluation is using it. To change a custom evaluator's criteria, clone it, edit the copy, and point the configuration at the copy. The original keeps the scoring history it already produced.

Where evaluators come from

An online evaluation runs the same catalog as on-demand evaluation (built-in, third-party, and custom evaluators, all through Amazon Bedrock AgentCore Evaluations), and you can mix kinds in one configuration. For what each kind covers and how to write your own, see Evaluators and evaluations.

The judge model depends on the evaluator kind, and that determines what you pay for. For built-in and third-party evaluators, the service selects and runs the judge model on its own capacity. There is nothing to configure. For custom evaluators, including one derived from a built-in or third-party evaluator, the judge model is the one you chose, and it is invoked with the permissions you grant the evaluation, so those model invocations run in your account. For the online-evaluation reference, see Evaluate agent performance, Online evaluation configurations, Prerequisites, and Results and output in the Amazon Bedrock AgentCore Developer Guide.

Permissions for custom evaluators

Built-in and third-party evaluators run on service-managed capacity — you grant nothing extra. A custom evaluator invokes the judge you chose, using the online evaluation's execution role, so that role must be allowed to invoke it. The permission depends on the judge:

• Amazon Bedrock foundation model or inference profile — bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream on that model's or inference profile's ARN.

• A model served through the bedrock-mantle endpoint in Amazon Bedrock — bedrock-mantle:CreateInference on the project and bedrock-mantle:CallWithBearerToken.

• Lambda (code) evaluator — lambda:InvokeFunction and lambda:GetFunction on your evaluator function.

The default execution role created for you covers built-in and third-party evaluators. When you add a custom evaluator, add the matching permission above to the execution role before you enable the evaluation; otherwise enabling fails with a permissions error (for example, "the execution role does not have permissions to invoke the specified Bedrock models"). For the managed policies that carry these actions and the conditions on them, see IAM policies to use CloudWatch Omni. For the execution-role policy reference, see Prerequisites.

To create an online evaluation

1. In your space, under Evaluate, choose Online evaluations, and then choose New evaluation.

2. Select the agents to evaluate, the evaluators to run, and the sampling percentage.

3. Grant the permissions the evaluation needs. Every online evaluation needs permission to write its results (CloudWatch Logs and metrics). If you include a custom evaluator, the evaluation's execution role must also be allowed to invoke that evaluator's judge — the exact permission depends on the judge's type (see Permissions for custom evaluators). Built-in and third-party evaluators need no additional permissions.

4. Enable the evaluation, and confirm its status shows as enabled. Scores from sampled traffic appear in the Evaluation dashboard as they accumulate.

Cost and data considerations

• Judge-based evaluators invoke a model for every sampled trace. Tune the sampling percentage to match your traffic volume and budget.

• Evaluators that use a judge model send the sampled trace content to that model. If your traces can contain sensitive data, see Protect sensitive data.

For scoring a specific trace or dataset on demand, see Evaluators and evaluations. When an online evaluation surfaces a regression, follow Tutorial: Fix a production quality regression to fix and verify it.