Developer Guide

Searching for Traces in the AWS X-Ray Console with Filter Expressions

When you choose a time period of traces to view in the X-Ray console, you might get more results than the console can display. In the top-right corner, the console shows the number of traces that it scanned and, whether there are more traces available. You can narrow the results to just the traces that you want to find by using a filter expression.

When you choose a node in the service map, the console constructs a filter expression based on the service name of the node, and the types of error present based on your selection. To find traces that show performance issues or that relate to specific requests, you can adjust the expression provided by the console, or create your own. If you add annotations with the X-Ray SDK, you can also filter based on the presence of an annotation key or the value of a key.


If you choose a relative time range in the service map, the console converts it to an absolute start and end time when you choose a node. To ensure that the traces for the node appear in the search results, and avoid scanning times when the node was not active, the time range only includes times when the node sent traces. If you want to search relative to the current time, you can switch back to a relative time range in the traces page and re-scan.

If there are still more results available than the console can show, the console shows you how many traces matched and the number of traces scanned. The percentage shown is the percentage of the selected time frame that was scanned. Narrow your filter expression further, or choose a shorter time frame, to ensure that you see all matching traces represented in the results.

To get the freshest results first, the console starts scanning at the end of the time range and works backwards. If there are a large number of traces, but few results, the console splits the time range into chunks and scans them in parallel. The progress bar shows the parts of the time range that have been scanned.

Using Filter Expressions with Groups

Groups are a collection of traces that are defined by a filter expression. You can use groups to generate additional service graphs and supply Amazon CloudWatch metrics. Groups are identified by their name or an Amazon Resource Name (ARN), and contain a filter expression. The service compares incoming traces to the expression and stores them accordingly.

You can create and modify groups by using the drop-down menu to the left of the filter expression search bar.


If the service encounters an error in qualifying a group, that group is no longer included in processing incoming traces and an error metric is recorded.

Filter Expression Syntax

Filter expressions can contain a keyword, a unary or binary operator, and a value for comparison.

keyword operator value

Different operators are available for different types of keyword. For example, responsetime is a number keyword and can be compared with operators related to numbers.

Example Requests where response time was more than 5 seconds

responsetime > 5

You can combine multiple expressions in a compound expression with the AND and OR operators.

Example Requests where the total duration was 5 to 8 seconds

duration >= 5 AND duration <= 8

Simple keywords and operators only find issues at the trace level. If an error occurs downstream, but is handled by your application and not returned to the user, a search for error will not find it.

To find traces with downstream issues, you can use the complex keywords service() and edge(). These keywords let you apply a filter expression to all downstream nodes, a single downstream node, or an edge between two nodes. For even more granularity, you can filter services and edges by type with the id() function.

Boolean Keywords

Boolean keywords are either true or false. Use these keywords to find traces that resulted in errors.

Boolean Keywords

  • ok – Response status code was 2XX Success.

  • error – Response status code was 4XX Client Error.

  • throttle – Response status code was 429 Too Many Requests.

  • fault – Response status code was 5XX Server Error.

  • partial – Request has incomplete segments.

  • inferred – Request has inferred segments.

  • first – Element is the first of an enumerated list.

  • last – Element is the last of an enumerated list.

  • remote – Root cause entity is remote.

  • root – Service is the entry point or root segment of a trace.

Boolean operators find segments where the specified key is true or false.

Boolean Operators

  • none – The expression is true if the keyword is true.

  • ! – The expression is true if the keyword is false.

  • =,!= – Compare the value of the keyword to the string true or false. Acts the same as the other operators but is more explicit.

Example Response status is 2XX OK


Example Response status is not 2XX OK


Example Response status is not 2XX OK

ok = false

Example Last enumerated fault trace has error name "deserialize"

rootcause.fault.entity { last and name = "deserialize" }

Example Requests with remote segments where coverage > 0.7 and service name "traces"

rootcause.responsetime.entity { remote and coverage > 0.7 and name = "traces" }

Example Requests with inferred segments where service type "AWS:DynamonDB"

rootcause.fault.service { inferred and name = traces and type = "AWS::DynamoDB" }

Example Requests that have a segment with the name "data-plane" as the root.

service("data-plane") {root = true and fault = true}

Number Keywords

Number keywords let you search for requests with a specific response time, duration, or response status.

Number Keywords

  • responsetime – Time that the server took to send a response.

  • duration – Total request duration, including all downstream calls.

  • http.status – Response status code.

  • index – Position of an element in an enumerated list.

  • coverage – Decimal percentage of entity response time over root segment response time. Applicable only for response time root cause entities.

Number keywords use standard equality and comparison operators.

Number Operators

  • =,!= – The keyword is equal to or not equal to a number value.

  • <,<=, >,>= – The keyword is less than or greater than a number value.

Example Response status is not 200 OK

http.status != 200

Example Request where the total duration was 5 to 8 seconds

duration >= 5 AND duration <= 8

Example Requests that completed successfully in under 3 seconds, including all downstream calls

ok !partial duration <3

Example Enumerated list entity has index > 5

rootcause.fault.service { index > 5 }

Example Requests where last entity has coverage > 0.8

rootcause.responsetime.entity { last and coverage > 0.8 }

String Keywords

String keywords let you find traces with specific text in the request headers, or user IDs.

String Keywords

  • http.url – Request URL.

  • http.method – Request method.

  • http.useragent – Request user agent string.

  • http.clientip – Requestor's IP address.

  • user – Value of user field on any segment in the trace.

  • name – The name of a service or exception.

  • type – Service type.

  • message – Exception message.

  • availabilityzone – Value of availabilityzone field on any segment in the trace.

  • – Value of instance ID field on any segment in the trace.

  • resource.arn – Value of resource ARN field on any segment in the trace.

String operators find values that are equal to or contain specific text. Values must always be specified in quotation marks.

String Operators

  • =,!= – The keyword is equal to or not equal to a number value.

  • CONTAINS – The keyword contains a specific string.

  • BEGINSWITH ,ENDSWITH – The keyword starts or ends with a specific string.

Example Http.url filter

http.url CONTAINS "/api/game/"

To test if a field exists on a trace, regardless of its value, check to see if it contains the empty string.

Example User filter

Find all traces with user IDs.

user CONTAINS ""

Example Select traces with a fault root cause that includes service named "Auth"

rootcause.fault.service { name = "Auth" }

Example Select traces with a response time root cause whose last service has a type of DynamoDB

rootcause.responsetime.service { last and type = "AWS::DynamoDB" }

Example Select traces with a fault root cause whose last exception has the message "Access Denied for account_id: 1234567890"

rootcause.fault.exception { last and message = "Access Denied for account_id: 1234567890"

Complex Keywords

Complex keywords let you find requests based on service name, edge name, or annotation value. For services and edges, you can specify an additional filter expression that applies to the service or edge. For annotations, you can filter on the value of an annotation with a specific key using boolean, number or string operators.

Complex Keywords

  • service(name) {filter} – Service with name name. Optional curly braces can contain a filter expression that applies to segments created by the service.

  • edge(source, destination) {filter} – Connection between services source and destination. Optional curly braces can contain a filter expression that applies to segments on this connection.

  • annotation.key – Value of annotation with field key. The value of an annotation can be a boolean, number, or string, so you can use any of those type's comparison operators. You cannot use this keyword in combination with the service or edge keywords.

  • json – JSON root cause object. See Getting Data from AWS X-Ray for steps to create JSON entities programmatically.

Use the service keyword to find traces for requests that hit a certain node on your service map.

Example Service filter

Requests that included a call to with a fault (500 series error).

service("") { fault }

You can exclude the service name to apply a filter expression to all nodes on your service map.

Example Service filter

Requests that caused a fault anywhere on your service map.

service() { fault }

The edge keyword applies a filter expression to a connection between two nodes.

Example Edge filter

Request where the service made a call to that failed with an error.

edge("", "") { error }

You can also use the ! operator with service and edge keywords to exclude a service or edge from the results of another filter expression.

Example Service and request filter

Request where the URL begins with and contains /v2/ but does not reach a service named

http.url BEGINSWITH "" AND http.url CONTAINS "/v2/" AND !service("")

For annotations, use the comparison operators that correspond to the type of value.

Example Annotation with string value

Requests with an annotation named gameid with string value "817DL6VO".

annotation.gameid = "817DL6VO"

Example Annotation with number value

Requests with annotation age with numerical value greater than 29.

annotation.age > 29

Example JSON with root cause entity

Requests with matching root cause entities.

rootcause.json = #[{ "Services": [ { "Name": "GetWeatherData", "EntityPath": [{ "Name": "GetWeatherData" }, { "Name": "get_temperature" } ] }, { "Name": "GetTemperature", "EntityPath": [ { "Name": "GetTemperature" } ] } ] }]

The ID Function

When you provide a service name to the service or edge keywords, you get results for all nodes that have that name. For more precise filtering, you can use the id function to specify a service type in addition to a name to distinguish between nodes with the same name.

id(name: "service-name", type:"service::type")

You can use the id function in place of a service name in service and edge filters.

service(id(name: "service-name", type:"service::type")) { filter }
edge(id(name: "service-one", type:"service::type"), id(name: "service-two", type:"service::type")) { filter }

For example, the Scorekeep sample application includes an AWS Lambda function named random-name. This creates two nodes in the service map, one for the function invocation, and one for the Lambda service.

The two nodes have the same name but different types. A standard service filter will find traces for both.

Example Service filter

Requests that include an error on any service named random-name.

service("random-name") { error }

Use the id function to narrow the search down to errors on the function itself, excluding errors from the service.

Example Service filter with id function

Requests that include an error on a service named random-name with type AWS::Lambda::Function.

service(id(name: "random-name", type: "AWS::Lambda::Function")) { error }

You can also exclude the name entirely, to search for nodes by type.

Example Service filter with id() function

Requests that include an error on a service with type AWS::Lambda::Function.

service(id(type: "AWS::Lambda::Function")) { error }