Enabling AgentCore runtime observability Enabling observability in agent code for AgentCore-hosted agents Configure Observability for agents hosted outside of the AgentCore runtime Enable observability for AgentCore memory, gateway, and built-in tool resources Enhanced AgentCore observability with custom headers Observability best practices Using other observability platforms

Add observability to your Amazon Bedrock AgentCore resources

Amazon Bedrock AgentCore provides a number of built-in metrics to monitor the performance of resources for the AgentCore runtime, memory, gateway, and built-in tool resource types. This default data is available in Amazon CloudWatch. To view the full range of observability data in the CloudWatch console, or to output custom runtime metrics for agents, you need to instrument your code using the AWS Distro for Open Telemetry (ADOT) SDK.

To view the observability dashboard in CloudWatch, open the Amazon CloudWatch GenAi Observability page.

See the following sections to learn more about configuring your resources to view observability metrics in the CloudWatch console generative AI observability page and in CloudWatch Logs.

Tip

Use of the ADOT SDK to output custom metrics is also supported for agents running outside the AgentCore runtime. To learn how to enable observability for these agents, see Configure Observability for agents hosted outside of the AgentCore runtime.

Enabling AgentCore runtime observability

To view metrics, spans, and traces generated by the AgentCore service, you first need to complete a one-time setup to turn on Amazon CloudWatch Transaction Search. To view service-provided spans for memory resources, you also need to enable tracing when you create a memory. See Enable observability for AgentCore memory, gateway, and built-in tool resources to learn more.

The following sections describe how to perform these setup actions and to enable observability in your agent code.

Enabling CloudWatch Transaction Search

You can enable CloudWatch Transaction Search either by using the CloudWatch console, or by using an API through the AWS Command Line Interface (AWS CLI) or one of the AWS SDKs.

Use one of the following procedures to enable Transaction Search.

CloudWatch console

To enable CloudWatch Transaction Search in the CloudWatch console

Open the CloudWatch console.
In the navigation pane, expand Application Signals (APM) and choose Transaction search.
Choose Enable Transaction Search.
Select the checkbox to ingest spans as structured logs.
(Optional) Change the percentage of spans you want to be indexed as trace summaries by entering a value under X-Ray trace indexing. By default, 1% of spans are indexed as trace summaries for free, but you can alter the percentage to generate more trace summaries for end-to-end transaction analysis.
Choose Save.

API

To enable CloudWatch Transaction Search using an API

When using the AWS CLI or an AWS SDK to enable Transaction Search, first configure the necessary permissions to ingest spans in CloudWatch Logs by adding a resource-based policy with PutResourcePolicy.

The following AWS CLI command adds a resource policy that gives AWS X-Ray permissions to send traces to CloudWatch Logs.


aws logs put-resource-policy --policy-name MyResourcePolicy --policy-document '{ "Version": "2012-10-17", "Statement": [ { "Sid": "TransactionSearchXRayAccess", "Effect": "Allow", "Principal": { "Service": "xray.amazonaws.com" }, "Action": "logs:PutLogEvents", "Resource": [ "arn:partition:logs:region:account-id:log-group:aws/spans:*", "arn:partition:logs:region:account-id:log-group:/aws/application-signals/data:*" ], "Condition": { "ArnLike": { "aws:SourceArn": "arn:partition:logs:region:account-id:*" }, "StringEquals": { "aws:SourceAccount": "account-id" } } } ]}'

For clarity, the inline JSON policy in this command is shown expanded in the following example:



{
    "Version": "2012-10-17",		 	 	 
    "Statement": [
        {
            "Sid": "TransactionSearchXRayAccess",
            "Effect": "Allow",
            "Principal": {
                "Service": "xray.amazonaws.com"
            },
            "Action": "logs:PutLogEvents",
            "Resource": [
                "arn:aws:logs:us-east-1:123456789012:log-group:aws/spans:*",
                "arn:aws:logs:us-east-1:123456789012:log-group:/aws/application-signals/data:*"
            ],
            "Condition": {
                "ArnLike": {
                    "aws:SourceArn": "arn:aws:xray:us-east-1:123456789012:*"
                },
                "StringEquals": {
                    "aws:SourceAccount": "123456789012"
                }
            }
        }
    ]
}

Configure the destination of your trace segments using UpdateTraceSegmentDestination.

To use the AWS CLI, run the following command.
```
aws xray update-trace-segment-destination --destination CloudWatchLogs
```
(Optional) Configure your desired sampling percentage using UpdateIndexingRule.

To use the AWS CLI, run the following command.
```
aws xray update-indexing-rule --name "Default" --rule '{"Probabilistic": {"DesiredSamplingPercentage": number}}'
```

Enabling observability in agent code for AgentCore-hosted agents

In addition to the service-generated metrics, with AgentCore you can also gather span and trace data as well as custom metrics emitted from your agent code.

When you use agent frameworks like Strands, LangChain, or CrewAI with supported third-party instrumentation libraries, the framework itself comes with built in support for OTEL and GenAI semantic conventions, and it can also be instrumented with an auto-instrumentation package such as opentelemetry-instrument-langchain. It is also possible to send Generative AI semantic conventions telemetry and spans by defining a custom tracer. AgentCore supports use of the following instrumentation libraries in your agent framework:

To view this data in the CloudWatch console generative AI observability page and in Amazon CloudWatch, you need to add the AWS Distro for Open Telemetry (ADOT) SDK to your agent code.

Note

With AgentCore, you can also view metrics for agents that aren't running in the AgentCore runtime. Additional setup steps are required to configure telemetry outputs for non-AgentCore agents. See the instructions in Configure Observability for agents hosted outside of the AgentCore runtime to learn more.

To add ADOT support and enable AgentCore observability, follow the steps in the following procedure.

Add observability to your AgentCore agent

Ensure that your framework is configured to emit traces. For example, in the Strands framework, the tracer object must be configured to instruct Strands to emit Open Telemetry (OTEL) logs.
Add the ADOT SDK and boto3 to your agent's dependencies. For Python, add the following to your requirements.txt file:
```
aws-opentelemetry-distro>=0.10.0
boto3
```
Alternatively, you can install the dependencies directly:
```
pip install aws-opentelemetry-distro>=0.10.0 boto3
```
Execute your agent code using the OpenTelemetry auto-instrumentation command:
```
opentelemetry-instrument python my_agent.py
```
This auto-instrumentation approach automatically adds the SDK to the Python path. You may already be using this approach as part of your standard OpenTelemetry implementation.

For containerized environment (such as docker) add the following command:
```
CMD ["opentelemetry-instrument", "python", "main.py"]
```
When using ADOT, in order to propagate session id correctly, define the X-Amzn-Bedrock-AgentCore-Runtime-Session-Id in the request header. ADOT then sets the session_id correctly in the downstream headers.

To propoagate a trace ID, invoke the AgentCore runtime with the parameter traceId=<traceId> set.

You can also invoke your agent with additional headers for additional observability options. See Enhanced AgentCore observability with custom headers to learn more.

Configure Observability for agents hosted outside of the AgentCore runtime

To enable observability for agents hosted outside of the AgentCore runtime, first follow the steps in the previous sections to enable CloudWatch Transaction Search and add the ADOT SDK to your code.

For agents running outside of the AgentCore runtime, you also need to create an agent log-group which you include in your environment variables.

Configure your AWS environment variables, and then set your Open Telemetry environment variables as shown in the following.

AWS environment variables


AWS_ACCOUNT_ID=<account id>
AWS_DEFAULT_REGION=<default region>
AWS_REGION=<region>
AWS_ACCESS_KEY_ID=<access key id>
AWS_SECRET_ACCESS_KEY=<secret key>

OTEL environment variables


AGENT_OBSERVABILITY_ENABLED=true
OTEL_PYTHON_DISTRO=aws_distro
OTEL_PYTHON_CONFIGURATOR=aws_configurator # required for ADOT Python only
OTEL_RESOURCE_ATTRIBUTES=service.name=<agent-name>,aws.log.group.names=/aws/bedrock-agentcore/runtimes/<agent-id>,cloud.resource_id=<AgentEndpointArn:AgentEndpointName> # endpoint is optional
OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=/aws/bedrock-agentcore/runtimes/<agent-id>,x-aws-log-stream=runtime-logs,x-aws-metric-namespace=bedrock-agentcore
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_TRACES_EXPORTER=otlp

Replace <agent-name> with your agent's name and <agent-id> with a unique identifier for your agent.

Session ID support

To propagate session ID, you need to invoke using session identifier in the OTEL baggage:


from opentelemetry import baggage 
 
ctx = baggage.set_baggage("session.id", session_id) # Set the session.id in baggage
attach(ctx) # Attach the context to make it active token

Enable observability for AgentCore memory, gateway, and built-in tool resources

When you create an AgentCore runtime resource (agent), by default, AgentCore runtime creates a CloudWatch log group for the service-provided logs. However, for memory, gateway, and built-in tool resources, AgentCore doesn't configure log destinations for you automatically.

For memory and gateway resources, you can configure log destinations either in the console or by using an AWS SDK. If you use the console to configure a CloudWatch Logs destination, the default log group name for memory and gateway resources has the form /aws/vendedlogs/bedrock-agentcore/{resource-type}/APPLICATION_LOGS/{resource-id}, where {resource-type} is either memory or gateway.

For memory and gateway logs, you can also configure log destinations in Amazon S3 logs or Firehose stream logs using the AgentCore console. To learn more about storing logs in Amazon S3 or Firehose, see Uploading, downloading, and working with objects in Amazon S3 and Creating an Amazon Data Firehose delivery stream.

To learn more about the log data output by AgentCore for memory and gateway resources see Provided log data (memory) or Provided log data (gateway).

For built-in tool resources, the AgentCore service doesn't provide logs by default, but you can output your own logs from your code. If you supply your own log outputs, you need to manually configure log destinations to store this data.

To see what observability data AgentCore provides by default for each resource type, see Amazon Bedrock AgentCore provided observability metrics.

Configure log destinations using the console

To configure log destinations for memory or gateway logs in the AgentCore console, use the following procedures.

Memory

To configure log delivery for memory resources (console)

Open the Memory page in the AgentCore console.
In the Memory pane, select the memory you want to configure a log destination for.
Scroll down to the Log delivery pane and choose Add.
From the dropdown list, select the type of log destination you want to add (CloudWatch Logs group, Amazon S3 bucket, or Amazon Data Firehose).
For Log type, select APPLICATION_LOGS.
For Amazon S3 and Firehose destinations, enter a Delivery destination ARN. For CloudWatch Logs, the Destination log group is already populated with a default value.
(Optional) For CloudWatch Logs destinations, to change the default log group, enter a new log group name or select an existing log group under Destination log group.
(Optional) To change the fields that are captured in each log record or the logs' output format, expand Additional settings - optional, and modify the Field selection, Output format, and Field delimiter to your desired configuration.
Choose Add.

Gateway

To configure log delivery for gateway resources (console)

Open the Gateways page in the AgentCore console.
In the Gateways pane, select the gateway you want to configure a log destination for.
Scroll down to the Log delivery pane and choose Add.
From the dropdown list, select the type of log destination you want to add (CloudWatch Logs group, Amazon S3 bucket, or Amazon Data Firehose).
For Amazon S3 and Firehose destinations, enter a Delivery destination ARN. For CloudWatch Logs, the Destination log group is already populated with a default value.
(Optional) For CloudWatch Logs destinations, to change the default log group, enter a new log group name or select an existing log group under Destination log group.
(Optional) To change the fields that are captured in each log record or the logs' output format, expand Additional settings - optional, and modify the Field selection, Output format, and Field delimiter to your desired configuration.
Choose Add.

Configure CloudWatch resources using an AWS SDK

To configure a delivery source for logs and traces (SDK)

Run the following Python code to configure CloudWatch for your memory, gateway, and built-in tool resources. Note that delivery sources and destinations for tracing are only applicable for memory resources.


import boto3

def enable_observability_for_resource(resource_arn, resource_id, account_id, region='us-east-1'):
    """
    Enable observability for a Bedrock AgentCore resource (e.g., Memory Store)
    """
    logs_client = boto3.client('logs', region_name=region)

    # Step 0: Create new log group for vended log delivery
    log_group_name = f'/aws/vendedlogs/bedrock-agentcore/{resource_id}'
    logs_client.create_log_group(logGroupName=log_group_name)
    log_group_arn = f'arn:aws:logs:{region}:{account_id}:log-group:{log_group_name}'
    
    # Step 1: Create delivery source for logs
    logs_source_response = logs_client.put_delivery_source(
        name=f"{resource_id}-logs-source",
        logType="APPLICATION_LOGS",
        resourceArn=resource_arn
    )
    
    # Step 2: Create delivery source for traces  
    traces_source_response = logs_client.put_delivery_source(
        name=f"{resource_id}-traces-source", 
        logType="TRACES",
        resourceArn=resource_arn
    )
    
    # Step 3: Create delivery destinations
    logs_destination_response = logs_client.put_delivery_destination(
        name=f"{resource_id}-logs-destination",
        deliveryDestinationType='CWL',
        deliveryDestinationConfiguration={
            'destinationResourceArn': log_group_arn,
        }
    )
    
    # Traces required for memory only
    traces_destination_response = logs_client.put_delivery_destination(
        name=f"{resource_id}-traces-destination",
        deliveryDestinationType='XRAY'
    )
    
    # Step 4: Create deliveries (connect sources to destinations)
    logs_delivery = logs_client.create_delivery(
        deliverySourceName=logs_source_response['deliverySource']['name'],
        deliveryDestinationArn=logs_destination_response['deliveryDestination']['arn']
    )
    
    # Traces required for memory only
    traces_delivery = logs_client.create_delivery(
        deliverySourceName=traces_source_response['deliverySource']['name'], 
        deliveryDestinationArn=traces_destination_response['deliveryDestination']['arn']
    )
    
    print(f"Observability enabled for {resource_id}")
    return {
        'logs_delivery_id': logs_delivery['id'],
        'traces_delivery_id': traces_delivery['id']
    }

# Usage example
resource_arn = "arn:aws:bedrock-agentcore:us-east-1:123456789012:memory/my-memory-id"
resource_id = "my-memory-id" 
account_id = "123456789012"

delivery_ids = enable_observability_for_resource(resource_arn, resource_id, account_id)

Enhanced AgentCore observability with custom headers

You can invoke your agent with additional HTTP headers to provide enhanced observability options. The following example shows invocations including optional additional header requests for agents hosted in the AgentCore runtime.

Example Boto3 invocation


def invoke_agent(agent_id, payload, session_id=None):
    client = boto3.client("bedrock-agentcore", region="us-west-2")
    response = client.invoke_agent_runtime(
        agentRuntimeArn="arn:aws:bedrock-agentcore:us-west-2:111122223333:runtime/test_agent_boto2-nIg2xk3VSR",
        runtimeSessionId="12345678-1234-5678-9abc-123456789012",
        payload='{"query": "Plan a weekend in Seattle"}',
    )

You can include the following optional headers when invoking your agent to enhance observability and tracing capabilities:

Optional request headers for observability
Header	Description	Sample Value	Technical Explanation
X-Amzn-Trace-Id	Trace ID for request tracking (X-Ray Format)	Root=1-5759e988-bd862e3fe1be46a994272793;Parent=53995c3f42cd8ad8;Sampled=1	Used for distributed tracing across AWS services. Contains root ID (request origin), parent ID (previous service), and sampling decision for tracing. Sampling=1 means 100% sampling. Parent is X-Ray Trace format as well. OTEL will auto-generate trace IDs if not supplied.
traceparent	W3C standard tracing header	00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01	W3C format that includes version, trace ID, parent ID, and flags. Required for cross-service trace correlation when using modern tracing systems.
X-Amzn-Bedrock-AgentCore-Runtime-Session-Id	AgentCore session identifier	a1b2c3d4-5678-90ab-cdef-EXAMPLEaaaaa	Identifies a user session within the AgentCore system. Helps with session-based analytics and troubleshooting.
mcp-session-id	MCP session identifier	mcp-a1b2c3d4-5678-90ab-cdef-EXAMPLEaaaaa	Identifies a session in the Managed Cloud Platform. Enables tracing of operations across the MCP ecosystem.
tracestate	Additional tracing state information	congo=t61rcWkgMzE,rojo=00f067aa0ba902b7	Vendor-specific tracing information. Conveys additional context for tracing systems beyond what's in traceparent.
baggage	Context propagation for distributed tracing	userId=alice,serverRegion=us-east-1	Key-value pairs that propagate user-defined properties across service boundaries for contextual logging and analysis.

Observability best practices

Consider the following best practices when implementing observability for agents in AgentCore:

Use consistent session IDs - When possible, reuse the same session ID for related requests to maintain context across interactions.
Implement distributed tracing - Use the provided headers to enable end-to-end tracing across your application components.
Add custom attributes - Enhance your traces and metrics with custom attributes that provide additional context for troubleshooting and analysis.
Monitor resource usage - Pay attention to memory usage metrics to optimize your agent's performance.
Set up alerts - Configure CloudWatch alarms to help notify you of potential issues before they impact your users.

Using other observability platforms

To integrate agents hosted in the AgentCore runtime with other observability platforms to capture and view telemetry outputs, set the following environment variable:


DISABLE_ADOT_OBSERVABILITY=true

Setting this variable to true unsets the AgentCore runtime's default ADOT environment variables, ensuring that none of the default ADOT configurations are set.

Warning Javascript is disabled or is unavailable in your browser.

To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

AgentCore Observability: Observe your agents and resources

Observability concepts