AWS X-Ray Concepts
AWS X-Ray receives data from services in the form of segments, groups segments with a common request into traces, and processes traces to generate a service graph and provide a visual representation of your application.
The compute resources running your application logic send data about the work that they do in the form of segments. A segment provides the name of the resource, details about the request, and details about the work done. For example, when an HTTP request reaches your application, it can record data about:
The host – host name, alias or IP address
The request – method, client address, path, user agent
The response – status, content
The work done – start and end times, subsegments
The X-Ray SDK gathers information from request and response headers, the code in your application, and metadata about the AWS resources on which it runs. You choose which data is collected by adding modifying your application configuration or code to instrument incoming requests, downstream requests, and AWS SDK clients.
If a request is forwarded to your application by a load balancer or other intermediary,
The client IP in the segment is taken from the
X-Forwarded-For header in the
request instead of the source IP in the IP packet. The client IP recorded for a forwarded
request can be forged so should not be trusted.
For details on the structure and information recorded in segments and subsegments, see AWS X-Ray Segment Documents.
Data about the work done can be broken down into subsegments, which provide more granular timing information and details about downstream calls that your application made to fulfill the original request. A subsegment can contain additional details about a call to an AWS service, an external HTTP API, or an SQL database. You can even define arbitrary subsegments to instrument specific functions or lines of code in your application.
X-Ray uses the data that your application sends to generate a service graph. Each AWS resource that sends data to X-Ray appears as a service, with edges connecting the services that work together to serve requests. Edges connect clients to your application, and your application to the downstream services and resources that it uses.
name should match the domain name or logical name of the service
generates the segment, but this is not enforced. Any application with permission to
PutTraceSegments can send segments with any name.
A service graph is a JSON document with information about the services and resources that comprise your application. The X-Ray console uses the service graph to generate a service map, a visualization of the service graph.
For a distributed application, X-Ray combines nodes from all services that process requests with the same trace ID into a single service graph. The first service that the request hits adds a tracing header that is propagated between the frontend and services that it calls.
For example, Scorekeep runs a web API that calls a microservice (an AWS Lambda function) to generate random name using a Node.js library. The X-Ray SDK for Java generates the trace ID and includes it in calls to Lambda. Lambda sends tracing data also passes the trace ID to the function itself. The X-Ray SDK for Node.js also uses the trace ID to send data, so nodes for the API, the Lambda service, and the Lambda function all appear as separate, but connected, nodes on the service graph.
The path of a request through your application is tracked with a trace ID. A trace collects all of the segments generated by a single request, typically an HTTP GET or POST request that travels through a load balancer, hits your application code, and generates downstream calls to other AWS services or external web APIs. A trace ID header is added to each HTTP request by the first supported service that it interacts with, and propagates downstream to track the latency, disposition, and other request data.
To ensure that tracing is efficient, while still providing a representative sample of the requests that your application serves, the first service that a request hits applies a sampling algorithm to determine which requests get traced. You can modify the default sampling rules and configure different sampling rates for different routes that your application serves with the X-Ray SDK.
All requests are traced up to a configurable minimum, after which a percentage of requests
are traced to avoid unnecessary cost. The sampling decision and trace ID, are added to HTTP
requests in tracing headers named
X-Amzn-Trace-Id. The tracing header is added to the request by the first
X-Ray-integrated service that it hits, read by the X-Ray SDK, and included in the
Example Tracing header with root trace ID and sampling decision
Tracing Header Security
A tracing header can originate from the X-Ray SDK, an AWS service, or the client
request. Your application can remove
X-Amzn-Trace-Id from incoming requests to
avoid issues caused by users adding trace IDs or sampling decisions to their
The tracing header can also contain a parent segment ID if the request originated from an instrumented application. For example, if your application calls a downstream HTTP web API with an instrumented HTTP client, the X-Ray SDK adds the ID of the segment for the original request to the tracing header of the downstream request. An instrumented application that serves the downstream request can record the parent segment ID to connect the two requests.
Example Tracing header with root trace ID, parent segment ID and sampling decision
Even with sampling, a complex application generates a lot of data. The AWS X-Ray console provides an easy-to-navigate view of the service graph. It shows health and performance information that helps you identify issues and opportunities for optimization in your application. For advanced tracing, you can drill down to traces for individual requests, or use filter expressions to find traces related to specific paths or users.
Annotations and Metadata
When you instrument your application, the X-Ray SDK records information about incoming and outgoing requests, AWS resources used, and itself. You can add other information to the segment document in the form of annotations and metadata.
Annotations are simple key-value pairs that are indexed
for use with filter expressions. Use annotations
to record data that you want to use to group traces in the console, or when calling the
Metadata are key-value pairs with values of any type, including objects and lists, but are not indexed. Use metadata to record data that you want stored in the trace but don't need to use for searching traces.
You can view annotations and metadata in the segment or subsegment details in the X-Ray console.