Menu
AWS Lambda
Developer Guide

Troubleshooting Lambda-Based Applications

A typical Lambda-based application consists of one or more functions triggered by events such as object uploads to Amazon S3, Amazon SNS notifications, and API actions. Once triggered, those functions usually call downstream resources such as DynamoDB tables or Amazon S3 buckets, or make other API calls. AWS Lambda leverages Amazon CloudWatch to automatically emit metrics and logs for all invocations of your function. However, this mechanism might not be convenient for tracing the event source that invoked your Lambda function, or for tracing downstream calls that your function made. For a complete overview of how tracing works, see AWS X-Ray.

Tracing Lambda-Based Applications with AWS X-Ray

Note

This is prerelease documentation for a feature in preview release. It is subject to change. There is no sign-up required to use this feature.

AWS X-Ray is an AWS service that allows you to detect, analyze, and optimize performance issues with your AWS Lambda applications. X-Ray collects metadata from the Lambda service and any upstream or downstream services that make up your application. X-Ray uses this metadata to generate a detailed service graph that illustrates performance bottlenecks, latency spikes, and other issues that impact the performance of your Lambda application.

After using the Lambda on the AWS X-Ray Service Map to identify a problematic resource or component, you can zoom in and view a visual representation of the request. This visual representation covers the time from when an event source triggers a Lambda function until the function execution has completed. X-Ray provides you with a breakdown of your function’s operations, such as information regarding downstream calls your Lambda function made to other services. In addition, X-Ray integration with Lambda provides you with visibility into the AWS Lambda service overhead. It does so by displaying specifics such as your request’s dwell time and number of invocations.

Note

Only services that currently integrate with X-Ray show as standalone traces, outside of your Lambda trace. For a list of services that currently support X-Ray, see X-Ray.

Setting Up AWS X-Ray with Lambda

Following, you can find detailed information on how to set up X-Ray with Lambda.

Before You Begin

To enable tracing on your Lambda function using the Lambda CLI, you must first add tracing permissions to your function's execution role. To do so, take the following steps:

  • Sign in to the IAM console at https://console.aws.amazon.com/iam/.

  • Find the execution role for your Lambda function.

  • Attach the following managed policy: AWSXrayWriteOnlyAccess

To learn more about these policies, see AWS X-Ray.

If you are changing the tracing mode to active using the Lambda console, tracing permissions are added automatically, as explained in the next section.

Tracing

The path of a request through your application is tracked with a trace ID. A trace collects all of the segments generated by a single request, typically an HTTP GET or POST request.

There are two modes of tracing for a Lambda function:

  • Pass Through: This is the default setting for all Lambda functions if you have added tracing permissions to your function's execution role. This approach means the Lambda function is only traced if X-Ray has been enabled on an upstream service, such as AWS Elastic Beanstalk.

  • Active: When a Lambda function has this setting, Lambda automatically samples invocation requests, based on the sampling algorithm specified by X-Ray.

    Note

    X-Ray applies a sampling algorithm to ensure that tracing is efficient, while still providing a representative sample of the requests that your application serves. The default sampling algorithm is 1 request per minute, with 5 percent of requests sampled past that limit.

You can change the tracing mode for your Lambda function by using either the Lambda Management Console or the Lambda CreateFunction or UpdateFunctionConfiguration API actions.

If you use the Lambda console, the following applies:

  • When you change a function's tracing mode to active, tracing permissions are automatically attached to the function's execution role. If you receive an error stating Lambda couldn't add the AWSLambdaTraceExecution policy to your function's execution role, sign in to the IAM console at https://console.aws.amazon.com/iam/ and manually add the policy.

  • To enable active tracing, go to the Advanced settings section under Configure function and select the Enable active tracing box.

If you use the Lambda CreateFunction or UpdateFunctionConfiguration API actions:

Note

You receive an error if you haven't added tracing permissions to your function's execution role. For more information, see Before You Begin.

  • If you want the tracing mode to be active, set the TracingConfig parameter's Mode property to Active. Again, any new function has its tracing mode set to PassThrough by default.

  • Any new or updated Lambda function has its $LATEST version set to the value you specify.

Lambda on the AWS X-Ray Service Map

X-Ray displays three types of nodes on the service map for requests served by Lambda:

  • Lambda service (AWS::Lambda) – This type of node represents the time the request spent in the Lambda service. Timing starts when Lambda first receives the request and ends when the request leaves the Lambda service.

  • Lambda function (AWS::Lambda::Function) – This type of node represents the Lambda function’s execution time.

  • Downstream service calls – In this type, each downstream service call from within the Lambda function is represented by a separate node.

In the diagram following, the nodes represent (from left to right): The Lambda service, the user function, and a downstream call to Amazon S3:

For more information, see Viewing the Service Map.

Lambda as an AWS X-Ray Trace

From the service map, you can zoom in to see a trace view of your Lambda function. The trace will display in-depth information regarding your function invocations, represented as segments and subsegments:

  • Lambda service segment – This segment represents different information depending on the event source used to invoke the function:

    • Synchronous and stream event sources – The service segment measures the time from when the Lambda service receives the request/event and ends when the request leaves the Lambda service (after the final invocation for the request is completed).

    • Asynchronous - The service segment represents the response time, that is, the time it took the Lambda service to return a 202 response to the client.

    The Lambda service segment can include two types of subsegments:

    • Dwell time (asynchronous invocations only) – Represents the time the function spends in the Lambda service before being invoked. This subsegment starts when the Lambda service receives the request/event and ends when the Lambda function is invoked for the first time.

    • Attempt – Represents a single invocation attempt, including any overhead introduced by the Lambda service. Examples of overhead are time spent initializing the function’s code and function execution time.

  • Lambda function segment - Represents execution time for the function for a given invocation attempt. It starts when the function handler starts executing and ends when the function terminates. This segment can include three types of subsegments:

    • Initialization - The time spent running the initialization code of the function, defined as the code outside the Lambda function handler or static initializers.

    • Downstream calls - Calls made to other AWS services from the Lambda function’s code.

    • Custom subsegments - Custom subsegments or user annotations that you can add to the Lambda function segment by using the X-Ray SDK.

Note

For each traced invocation, Lambda emits the Lambda service segment and all of its subsegments. These segments are emitted regardless of the runtime and require no code changes.

Emitting Trace Segments from a Lambda Function

For each traced invocation, Lambda will emit the Lambda service segment and all of its subsegments. In addition, Lambda will emit the Lambda function segment and the init subsegment. These segments will be emitted regardless of the function’s runtime, and with no code changes or additional libraries required. If you want your Lambda function’s X-Ray traces to include custom segments, annotations, or subsegments for downstream calls, you might need to include additional libraries and annotate your code.

Note

Currently, these additional capabilities (custom segments, annotations, and subsegments for downstream calls) are only available for the Node.js and Java runtimes.

Node.js

In Node.js, you can have Lambda emit subsegments to X-Ray to show you information about downstream calls to other AWS services made by your function. To do so, you first need to include the the AWS X-Ray SDK for Node.js in your deployment package. In addition, wrap your AWS SDK require statement in the following manner:

Copy
var AWSXRay = require(‘aws-xray-sdk-core’); var AWS = AWSXRay.captureAWS(require(‘aws-sdk’));

Then, use the AWS variable defined in the example preceding to initialize any service client that you want to trace with X-Ray, for example:

Copy
s3Client = AWS.S3();

After following these steps, any call made from your function using s3Client result in an X-Ray subsegment that represents that call. As an example, you can run the Node.js function following to see how the trace looks in X-Ray:

Copy
var AWSXRay = require('aws-xray-sdk-core'); var AWS = AWSXRay.captureAWS(require('aws-sdk')); s3 = new AWS.S3({signatureVersion: 'v4'}); exports.handler = (event, context, callback) => { var params = {Bucket: BUCKET_NAME, Key: BUCKET_KEY, Body: BODY}; s3.putObject(params, function(err, data) { if (err) { console.log(err) } else { console.log(‘success!’) } }); };

Following is what a trace emitted by the code preceding looks like (synchronous invocation):

Java

In Java, you can have Lambda emit subsegments to X-Ray to show you information regarding downstream calls to other AWS services made by your function. To take advantage of this capability, include the The AWS X-Ray SDK for Java in your deployment package. No code changes are needed. As long as you are using an AWS SDK version 1.11.48 and preceding, there is no need to add any additional code lines for downstream calls from your function to be traced.

The AWS SDK will dynamically import the X-Ray SDK to emit subsegments for downstream calls made by your function. By using the X-Ray SDK for Java, you can instrument your code in order to emit custom subsegments and or add annotatations to your X-Ray segments.

The following example uses the X-Ray SDK for Java to instrument a Lambda function to emit a custom subsegment and send custom annotation to X-Ray:

Copy
package uptime; import java.io.IOException; import java.time.Instant; import java.util.HashMap; import java.util.Map; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.http.HttpResponse; import org.apache.http.client.HttpClient; import org.apache.http.client.methods.HttpGet; import com.amazonaws.regions.Regions; import com.amazonaws.services.dynamodbv2.AmazonDynamoDB; import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder; import com.amazonaws.services.dynamodbv2.model.AttributeValue; import com.amazonaws.services.lambda.runtime.Context; import com.amazonaws.xray.AWSXRay; import com.amazonaws.xray.proxies.apache.http.HttpClientBuilder; public class Hello { private static final Log logger = LogFactory.getLog(Hello.class); private static final AmazonDynamoDB dynamoClient; private static final HttpClient httpClient; static { dynamoClient = AmazonDynamoDBClientBuilder.standard().withRegion(Regions.US_EAST_1).build(); httpClient = HttpClientBuilder.create().build(); } public void checkUptime(Context context) { AWSXRay.createSubsegment("makeRequest", (subsegment) -> { HttpGet request = new HttpGet("https://aws.amazon.com/"); boolean is2xx = false; try { HttpResponse response = httpClient.execute(request); is2xx = (response.getStatusLine().getStatusCode() / 100) == 2; subsegment.putAnnotation("responseCode", response.getStatusLine().getStatusCode()); } catch (IOException ioe) { logger.error(ioe); } Map<String, AttributeValue> item = new HashMap<>(); item.put("Timestamp", new AttributeValue().withN("" + Instant.now().getEpochSecond())); item.put("2xx", new AttributeValue().withBOOL(is2xx)); dynamoClient.putItem("amazon-2xx", item); }); } }

Following is what a trace emitted by the code preceding looks like (synchronous invocation):

The AWS X-Ray Daemon in the Lambda Environment

The AWS X-Ray Daemon is a software application that gathers raw segment data and relays it to the AWS X-Ray service. The daemon works in conjunction with the AWS X-Ray SDKs so that data sent by the SDKs can reach the X-Ray service.

When you trace your Lambda function, the X-Ray daemon automatically runs in the Lambda environment to gather trace data and send it to X-Ray. When tracing, the X-Ray daemon consumes a maximum of 16 MB or 3 percent of your function’s memory allocation. For example, if you allocate 128 MB of memory to your Lambda function, the X-Ray daemon has 16 MB of your function’s memory allocation. If you allocate 1024 MB to your Lambda function, the X-Ray daemon has 31 MB allocated to it (3 percent). For more information, see The AWS X-Ray Daemon.

Note

Lambda will try to terminate the X-Ray daemon to avoid exceeding your function’s memory limit. For example, assume you have allocated 128 MB to your Lambda function, which means the X-Ray daemon will have 16 MB allocated to it. That leaves your Lambda function with a memory allocation of 112 MB. However, if your function exceeds 112 MB, the X-Ray daemon will be terminated to avoid throwing an out-of-memory error.

Using Environment Variables to Communicate with AWS X-Ray

AWS Lambda automatically generates three environment variables to facilitate communication with the X-Ray daemon, and set the configuration of the X-Ray SDK:

  • _X_AMZN_TRACE_ID: Contains the tracing header, which includes the sampling decision, trace ID, and parent segment ID. (To learn more about these properties, see Tracing Header.) If Lambda receives a tracing header when your function is invoked, that header will be used to populate the _X_AMZN_TRACE_ID environment variable. If a tracing header was not received, Lambda will generate one for you.

  • AWS_XRAY_CONTEXT_MISSING: The X-Ray SDK uses this variable to determine its behavior in the event that your function tries to record X-Ray data, but a tracing header is not available. Lambda sets this value to LOG_ERROR by default.

  • AWS_XRAY_DAEMON_ADDRESS: This environment variable exposes the X-Ray daemon’s address in the following format: IP_ADDRESS:PORT. You can use the X-Ray daemon’s address to send trace data to the X-Ray daemon directly, without using the X-Ray SDK.

Lambda Traces in the AWS X-Ray Console: Examples

The following shows Lambda traces for two different Lambda functions. Each trace showcases a trace structure for a different invocation type: asynchronous and synchronous.

  • Async - The example following shows an asynchronous Lambda request with one successful invocation and one downstream call to DynamoDB.

    The Lambda service segment encapsulates the response time, which is the time it took to return a response (for example, 202) to the client. It includes subsegments for the time spent in the Lambda service queue (dwell time) and each invocation attempt. (Only one invocation attempt appears in the example preceding.) Each attempt subsegment in the service segment will have a corresponding user function segment. In this example, the user function segment contains two subsegments: the initialization subsegment representing the function’s initialization code that is run before the handler, and a downstream call subsegment representing a ListTables call to DynamoDB.

    Status codes and error messages are displayed for each Invocation subsegment and for each downstream call.

  • Synchronous - The example following shows a synchronous request with one downstream call to Amazon S3.

    The Lambda service segment captures the entire time the request spends in the Lambda service. The service segment will have a corresponding User function segment. In this example, the User function segment contains a subsegment representing the function’s initialization code (code run before the handler), and a subsegment representing the PutObject call to Amazon S3.

Note

If you want to trace HTTP calls, you need to use an HTTP client. For more information, see Tracing Calls to Downstream HTTP Web Services with the X-Ray SDK for Java or Tracing Calls to Downstream HTTP Web Services with the X-Ray SDK for Node.js .