Using capacity provider system logs
Capacity provider system logs record what Lambda does on your behalf. This topic explains how to find the logs, how to filter them, and how to read and correlate each type of log. For the full list of events and metadata fields, see Capacity provider system log event reference.
Finding and querying system logs
Lambda sends system logs to the capacity provider's log group in CloudWatch Logs. To find the log group and view the logs, see Viewing CloudWatch logs for capacity providers.
Each log entry is a JSON object, so you can select events by any field. Use the operation field to choose a type of log, and the metadata fields to narrow down to a function, an instance, or an execution environment.
To search the logs from the AWS CLI, use filter-log-events with a filter pattern. The following example returns the warnings and errors from the last hour:
aws logs filter-log-events \ --log-group-name /aws/lambda/capacity-provider/my-capacity-provider\ --start-time $(($(date +%s) - 3600))000 \ --filter-pattern '{ $.level = "WARN" || $.level = "ERROR" }'
To analyze the logs, for example to count or group events, use CloudWatch Logs Insights. The following query lists the most recent events for one function:
fields @timestamp, level, operation, message
| filter metadata.functionName = "my-function"
| sort @timestamp desc
| limit 100
Some events apply to the whole capacity provider instead of one function, for example instance launches and the capacity provider vCPU limit. These events don't include functionName. To include them when you filter by function, add a NOT EXISTS condition:
{ $.metadata.functionName = "my-function" || $.metadata.functionName NOT EXISTS }
FunctionLifecycle logs
FunctionLifecycle logs record each function version on the capacity provider as it's activated, has its scaling configuration updated, is deactivated, and is deleted.
-
Select the events for a function version: Filter on
operationandfunctionVersionArn. To follow a version from start to finish, sort the events by time.{ $.operation = "FunctionLifecycle" && $.metadata.functionVersionArn = "arn:aws:lambda:us-east-1:111122223333:function:my-function:1" } -
Read the state: Each event includes
lastUpdateStatus(InProgress,Successful, orFailed). -
Read failures: Failures are logged as errors.
Function version activation failed.includes the reason code instateReasonCode, andFunction scaling configuration update failed.includes it inlastUpdateStatusReasonCode. The reason code tells you which logs to check next:{ $.operation = "FunctionLifecycle" && $.level = "ERROR" && $.metadata.functionVersionArn = "arn:aws:lambda:us-east-1:111122223333:function:my-function:1" }-
A reason code that starts with
FunctionError.means that the execution environments failed to initialize. TheExecutionEnvironmentLifecyclelogs for the function version show each failure and itsreasonCode. -
A reason code for an AWS KMS key, subnet, or security group means that a capacity provider setting caused the failure. The event also includes that setting in
kmsKeyArn,subnetIds, orsecurityGroupIds. For details, see Capacity provider settings in failure events. -
A reason code about capacity, such as
CapacityProviderScalingLimitExceeded, means that Lambda couldn't get enough capacity. TheAutoscalingandInstanceLifecyclelogs show the limit that was reached or the instance launches that failed.
-
-
See capacity that's still needed: While a function version waits for capacity,
No available capacity: requesting new instance launch.events includeexecutionEnvironmentDeficit, the number of execution environments that still need an instance. -
Correlate with other logs: A
FunctionLifecyclefailure is often the result of earlier events. Look at theInstanceLifecycle,Autoscaling, andExecutionEnvironmentLifecycleevents for the same capacity provider and function in the minutes before the failure.
ExecutionEnvironmentLifecycle logs
ExecutionEnvironmentLifecycle logs record each execution environment that Lambda creates and terminates for a function version on the capacity provider's instances.
-
Select the events for a function version: Filter on
operationandfunctionVersionArnto see every execution environment that Lambda created, terminated, or failed for that version.{ $.operation = "ExecutionEnvironmentLifecycle" && $.metadata.functionVersionArn = "arn:aws:lambda:us-east-1:111122223333:function:my-function:1" } -
Follow one execution environment: All events for one execution environment share its
cloudWatchLogStreamName. To narrow the results to one execution environment, add a condition on that field.{ $.operation = "ExecutionEnvironmentLifecycle" && $.metadata.cloudWatchLogStreamName = "log-stream-name" } -
Open the function's logs:
cloudWatchLogStreamNameis also the name of the log stream that contains the function's own logs from that execution environment. Open it in the function's log group to see output and errors from your code, such as initialization errors. -
Read failures: Failures are logged as errors.
Execution environment failed.events includereasonCode. For the possible values, see Execution environment failure reason codes. The initialization timeout event includesinitElapsedTimeMsandinitConfiguredTimeoutMs.{ $.operation = "ExecutionEnvironmentLifecycle" && $.level = "ERROR" && $.metadata.functionVersionArn = "arn:aws:lambda:us-east-1:111122223333:function:my-function:1" } -
Find the hosting instance:
instanceIdis the Amazon EC2 instance that hosts the execution environment. Use it to find the relatedInstanceLifecycleevents for that instance.
Autoscaling logs
Autoscaling logs record the scaling decisions that Lambda makes for each function version, and the limits that stop it from scaling.
-
See scaling decisions:
Scale-out triggeredandScale-in triggeredevents includeaction, theresourcesthat drove the decision, and the current utilization of each resource.{ $.operation = "Autoscaling" && $.level = "INFO" && ($.metadata.action = "scaleOut" || $.metadata.action = "scaleIn") } -
Find what blocked scaling: Scaling limits are logged as warnings. Each warning names the limit and includes its current value, for example
maxExecutionEnvironmentsandexecutionEnvironmentCount, orcurrentVCPUandvCPULimit.{ $.operation = "Autoscaling" && $.level = "WARN" } -
Include capacity provider events: The vCPU limit and scale-to-zero events apply to the whole capacity provider and don't include function fields. Include them with the
NOT EXISTScondition described in Finding and querying system logs. -
See scaling over time: The following CloudWatch Logs Insights query counts each type of scaling event in 5-minute intervals:
filter operation = "Autoscaling" | stats count(*) by bin(5m), message
InstanceLifecycle logs
InstanceLifecycle logs record the Amazon EC2 instances that Lambda launches, terminates, and replaces for the capacity provider. These events apply to the whole capacity provider and don't include function fields.
-
Follow one launch attempt: The
Launching instanceevent and theLaunchedorFailed to launchevent for the same attempt share alaunchRequestId. The following CloudWatch Logs Insights query shows the first and last event of each launch attempt:filter operation = "InstanceLifecycle" and ispresent(metadata.launchRequestId) | stats earliest(message) as firstEvent, latest(message) as lastEvent, min(@timestamp) as startTime, max(@timestamp) as endTime by metadata.launchRequestId -
Read launch failures: A
Failed to launchevent includes the error message from Amazon EC2 and the Amazon EC2 error code inerrorCode. For subnet and security group errors, it also includessubnetIdsorsecurityGroupIds. -
Follow one instance: After launch, events for an instance include its
instanceId. Filter oninstanceIdto see when the instance launched, failed a health check, and terminated.{ $.operation = "InstanceLifecycle" && $.metadata.instanceId = "i-0123456789abcdef0" }