Custom multi-dimension evaluators

A multi-dimension evaluator is a custom LLM-as-a-judge evaluator that rates one target on several named dimensions in a single judge call. Each dimension has its own instruction and its own rating scale, and produces its own result.

For example, a customer-support agent’s reply can be rated on issue resolution, policy compliance, and tone. Instead of creating three evaluators that each read the same trace, you create one evaluator with three dimensions. The judge model reads the trace once and returns three results.

Use a multi-dimension evaluator when:

• You rate the same target on several dimensions and want one judge call instead of several.

• You want to mix scored dimensions (numerical) and classification dimensions (categorical) in one evaluator.

Use separate evaluators instead when:

• The dimensions need different evaluation levels or different judge models.

• You need the evaluator in an A/B test or a recommendation. Multi-dimension evaluators aren’t supported there.

• You don’t want a single failed judge call to fail every dimension. See Errors and token usage.

Define a multi-dimension evaluator

To create a multi-dimension evaluator, set llmAsAJudge.ratingScale.multiDimensional to a list of dimensions when you call CreateEvaluator. ratingScale takes exactly one of numerical, categorical, or multiDimensional.

Each dimension has the following fields. All fields are required.

Field	Description
dimensionName	The name of the dimension. Must start with a letter and contain only letters, digits, and underscores, up to 48 characters. Must be unique within the evaluator. The name identifies the dimension’s results.
dimensionInstruction	What the judge model assesses for this dimension, 1–4,000 characters. It can use the same placeholders as instructions for the evaluator’s level, but doesn’t have to include one.
dimensionRatingScale	The scale for this dimension: exactly one of numerical or categorical. Each is defined the same way as the numerical or categorical rating scale of a single-scale evaluator.

For the maximum number of dimensions, see Quotas for Amazon Bedrock AgentCore. Very long instructions across many dimensions can make the evaluator too large to store, and CreateEvaluator or UpdateEvaluator fails.

Multi-dimension evaluators support all three evaluation levels (TOOL_CALL, TRACE, and SESSION) and customer managed keys in AWS Key Management Service (AWS KMS). For encryption details, see Encryption at rest for AgentCore Evaluations.

The following configuration defines a trace-level evaluator with two numerical dimensions and one categorical dimension.

{
    "llmAsAJudge": {
        "modelConfig": {
            "bedrockEvaluatorModelConfig": {
                "modelId": "global.anthropic.claude-sonnet-4-5-20250929-v1:0",
                "inferenceConfig": {
                    "maxTokens": 2000,
                    "temperature": 0
                }
            }
        },
        "instructions": "You are reviewing a customer support assistant's reply. Rate the reply on each dimension below.\n\nContext: {context}\nCandidate Response: {assistant_turn}",
        "ratingScale": {
            "multiDimensional": [
                {
                    "dimensionName": "issue_resolution",
                    "dimensionInstruction": "Does the reply resolve the customer's issue, or give a clear next step that will resolve it?",
                    "dimensionRatingScale": {
                        "numerical": [
                            {"value": 1, "label": "Resolved", "definition": "The issue is resolved or a correct next step is given."},
                            {"value": 0.5, "label": "Partial", "definition": "The reply addresses part of the issue."},
                            {"value": 0, "label": "Unresolved", "definition": "The reply does not address the issue."}
                        ]
                    }
                },
                {
                    "dimensionName": "policy_compliance",
                    "dimensionInstruction": "Does the reply follow the support policy: no refunds promised without an order ID, no account details disclosed?",
                    "dimensionRatingScale": {
                        "numerical": [
                            {"value": 1, "label": "Compliant", "definition": "The reply follows the policy."},
                            {"value": 0, "label": "Violation", "definition": "The reply breaks the policy."}
                        ]
                    }
                },
                {
                    "dimensionName": "tone",
                    "dimensionInstruction": "Classify the tone of the reply.",
                    "dimensionRatingScale": {
                        "categorical": [
                            {"label": "friendly", "definition": "Warm and polite."},
                            {"label": "neutral", "definition": "Polite but impersonal."},
                            {"label": "curt", "definition": "Short, dismissive, or impolite."}
                        ]
                    }
                }
            ]
        }
    }
}

Create the evaluator the same way as any custom evaluator. For example, with the AWS SDK for Python (Boto3):

import json
import boto3

client = boto3.client("bedrock-agentcore-control")

with open("support_quality_config.json") as f:
    evaluator_config = json.load(f)

response = client.create_evaluator(
    evaluatorName="SupportQuality",
    level="TRACE",
    evaluatorConfig=evaluator_config,
)
print(response["evaluatorId"])

In the console, choose Multiple dimensions when you create the evaluator. For the steps, see Create evaluator.

The AgentCore CLI --rating-scale option takes a single numerical or categorical scale. To create a multi-dimension evaluator, use the console, the AWS SDK, or the AWS CLI.

Write instructions for dimensions

The judge model receives the evaluator’s instructions, then a section for each dimension with its name, instruction, and rating scale. The service then adds the output format, which asks for a rating and reasoning for every dimension. Don’t describe an output format in your instructions. To get consistent results:

• Put shared content in instructions: the task, the placeholders that bring in the trace or session, and how to read the conversation. instructions must include at least one placeholder for the evaluator’s level, as for any LLM-as-a-judge evaluator.

• Keep each dimensionInstruction to the one thing that dimension rates. Don’t repeat the shared context.

• Make dimensions independent. If two dimensions measure the same thing, merge them.

• The judge model writes an explanation and a rating for every dimension, so its output grows with the number of dimensions. Set maxTokens (or maxOutputTokens for responsesEvaluatorModelConfig) high enough for all dimensions. If the output is cut off, the evaluation fails.

Results

A multi-dimension evaluator produces one result per dimension for each target it evaluates. The dimension is identified as follows on each surface.

Surface	Result format
On-demand evaluation (Evaluate)	One entry in evaluationResults per dimension. evaluatorName is the evaluator’s name, and evaluationDimensionName is the dimension’s name. Numerical dimensions return value and label, and categorical dimensions return label.
Online and batch evaluation (CloudWatch Logs)	One result record per dimension. The gen_ai.evaluation.name attribute is <evaluatorName>/<dimensionName>, for example SupportQuality/tone.
CloudWatch Metrics	One metric per numerical dimension, named <evaluatorName>/<dimensionName>, with the same metric dimensions as other evaluators. Categorical dimensions don’t publish metrics. Their results are in CloudWatch Logs.
Batch evaluation	One entry in evaluatorSummaries per dimension, with evaluationDimensionName set. For categorical dimensions, statistics is empty. Sessions that fail before the judge runs are counted in a separate entry for the evaluator without evaluationDimensionName. See Errors and token usage.

The following is an excerpt of an Evaluate response for the SupportQuality evaluator on one trace:

{
    "evaluationResults": [
        {
            "evaluatorId": "SupportQuality-a1b2c3d4e5",
            "evaluatorName": "SupportQuality",
            "evaluationDimensionName": "issue_resolution",
            "value": 1.0,
            "label": "Resolved",
            "explanation": "The assistant identified the missing order and gave the correct tracking link...",
            "context": {"spanContext": {"sessionId": "session-123", "traceId": "6a827b653032d37116d7c1931a9835b5"}},
            "tokenUsage": {"inputTokens": 1015, "outputTokens": 316, "totalTokens": 1331}
        },
        {
            "evaluatorId": "SupportQuality-a1b2c3d4e5",
            "evaluatorName": "SupportQuality",
            "evaluationDimensionName": "policy_compliance",
            "value": 1.0,
            "label": "Compliant",
            "explanation": "No refund was promised and no account details were shared...",
            "context": {"spanContext": {"sessionId": "session-123", "traceId": "6a827b653032d37116d7c1931a9835b5"}}
        },
        {
            "evaluatorId": "SupportQuality-a1b2c3d4e5",
            "evaluatorName": "SupportQuality",
            "evaluationDimensionName": "tone",
            "label": "friendly",
            "explanation": "The reply is warm and thanks the customer...",
            "context": {"spanContext": {"sessionId": "session-123", "traceId": "6a827b653032d37116d7c1931a9835b5"}}
        }
    ]
}

To query one dimension’s online results in CloudWatch Logs Insights, filter on the composite name:

fields @timestamp, attributes.gen_ai.evaluation.score.value, attributes.gen_ai.evaluation.score.label
| filter attributes.gen_ai.evaluation.name = "SupportQuality/policy_compliance"
| sort @timestamp desc

For more about result records and metrics, see Results and output.

Errors and token usage

• Evaluation-level errors. If the evaluation can’t run for a target:

  • In on-demand evaluation, every dimension returns an error.

  • In online and batch evaluation, you get a single error result named after the evaluator, without a dimension. In CloudWatch Logs, its gen_ai.evaluation.name is the evaluator name alone. In batch evaluatorSummaries, these sessions are counted in totalFailed of an entry for the evaluator that has no evaluationDimensionName.

• Judge call errors. If the judge call fails or its output can’t be parsed, for example because it was cut off at maxTokens, every dimension returns an error.

• Dimension errors. If the judge returns a missing or out-of-scale rating for a dimension, the service asks the judge to correct it. If the rating is still invalid, only that dimension returns an error. The other dimensions keep their results.

• Token usage. Token usage is reported once per judge call, in tokenUsage on the first dimension that was scored successfully. If every dimension returned an error, no result includes tokenUsage. To get the token usage for a target, add up tokenUsage across that target’s results.

Feature support

Feature	Supported	Notes
On-demand evaluation	Yes	One result per dimension.
Online evaluation	Yes	One result record per dimension.
Batch evaluation	Yes	One summary per dimension.
A/B testing	No	Using a multi-dimension evaluator in an A/B test returns a validation error. See A/B testing prerequisites.
Recommendations	No	A recommendation whose evaluator is a multi-dimension evaluator returns a validation error. See System prompt recommendations.