Setting up CloudWatch logging for a REST API in API Gateway
To help debug issues related to request execution or client access to your API, you can enable Amazon CloudWatch Logs to log API calls. For more information about CloudWatch, see Monitoring REST API execution with Amazon CloudWatch metrics.
CloudWatch log formats for API Gateway
There are two types of API logging in CloudWatch: execution logging and access logging. In execution logging, API Gateway manages the CloudWatch Logs. The process includes creating log groups and log streams, and reporting to the log streams any caller's requests and responses.
The logged data includes errors or execution traces (such as request or response parameter values or payloads), data used by Lambda authorizers (formerly known as custom authorizers), whether API keys are required, whether usage plans are enabled, and so on.
When you deploy an API, API Gateway creates a log group and log streams under the log group.
The log group is named following the
API-Gateway-Execution-Logs_{rest-api-id}/{stage_name}
format. Within
each log group, the logs are further divided into log streams, which are ordered by
Last Event Time as logged data is reported.
In access logging, you, as an API developer, want to log who has accessed your API and how the caller accessed
the API. You can create your own log group or choose an existing log group that could be managed by API Gateway. To
specify the access details, you select $context
variables (expressed in a format of your choosing) and choose a log group as the destination. The access log
format must include at least $context.requestId
or $context.extendedRequestId
. As a best
practice, include $context.requestId
and $context.extendedRequestId
in your log format.
$context.requestId
logs the value in the x-amzn-RequestId
header. Clients can override
the value in the x-amzn-RequestId
header with a value in the format of a universally unique
identifier (UUID). API Gateway returns this request ID in the x-amzn-RequestId
response header. API Gateway
replaces overridden request IDs that aren't in the format of a UUID with
in your access logs.
UUID
_REPLACED_INVALID_REQUEST_ID$context.extendedRequestId
is a unique ID that API Gateway generates. API Gateway returns this request ID in
the x-amz-apigw-id
response header. An API caller can't provide or override this request ID. For more
information, see $context Variables for data
models, authorizers, mapping templates, and CloudWatch access logging.
Note
Only $context
variables are supported (not $input
, and
so on).
Choose a log format that is also adopted by your analytic backend, such as Common Log Format
Examples of some commonly used access log formats are shown in the API Gateway console and are listed as follows.
-
CLF
(Common Log Format): $context.identity.sourceIp $context.identity.caller \ $context.identity.user [$context.requestTime] \ "$context.httpMethod $context.resourcePath $context.protocol" \ $context.status $context.responseLength $context.requestId $context.extendedRequestId
The continuation characters (
\
) are meant as a visual aid. The log format must be a single line. You can add a newline character (\n
) at the end of the log format to include a newline at the end of each log entry. -
JSON
:{ "requestId":"$context.requestId", \ "extendedRequestId":"$context.extendedRequestId", \ "ip": "$context.identity.sourceIp", \ "caller":"$context.identity.caller", \ "user":"$context.identity.user", \ "requestTime":"$context.requestTime", \ "httpMethod":"$context.httpMethod", \ "resourcePath":"$context.resourcePath", \ "status":"$context.status", \ "protocol":"$context.protocol", \ "responseLength":"$context.responseLength" \ }
The continuation characters (
\
) are meant as a visual aid. The log format must be a single line. You can add a newline character (\n
) at the end of the log format to include a newline at the end of each log entry. -
XML
:<request id="$context.requestId"> \ <extendedRequestId>$context.extendedRequestId</extendedRequestId> <ip>$context.identity.sourceIp</ip> \ <caller>$context.identity.caller</caller> \ <user>$context.identity.user</user> \ <requestTime>$context.requestTime</requestTime> \ <httpMethod>$context.httpMethod</httpMethod> \ <resourcePath>$context.resourcePath</resourcePath> \ <status>$context.status</status> \ <protocol>$context.protocol</protocol> \ <responseLength>$context.responseLength</responseLength> \ </request>
The continuation characters (
\
) are meant as a visual aid. The log format must be a single line. You can add a newline character (\n
) at the end of the log format to include a newline at the end of each log entry. -
CSV
(comma-separated values):$context.identity.sourceIp,$context.identity.caller,\ $context.identity.user,$context.requestTime,$context.httpMethod,\ $context.resourcePath,$context.protocol,$context.status,\ $context.responseLength,$context.requestId,$context.extendedRequestId
The continuation characters (
\
) are meant as a visual aid. The log format must be a single line. You can add a newline character (\n
) at the end of the log format to include a newline at the end of each log entry.
Permissions for CloudWatch logging
To enable CloudWatch Logs, you must grant API Gateway permission to read and write logs to CloudWatch for
your account. The AmazonAPIGatewayPushToCloudWatchLogs
managed policy (with
an ARN of
arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs
)
has all the required permissions:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:PutLogEvents", "logs:GetLogEvents", "logs:FilterLogEvents" ], "Resource": "*" } ] }
Note
API Gateway calls AWS Security Token Service in order to assume the IAM role, so make sure that AWS STS is enabled for the Region. For more information, see Managing AWS STS in an AWS Region.
To grant these permissions to your account, create an IAM role with
apigateway.amazonaws.com
as its trusted entity, attach the preceding
policy to the IAM role, and set the IAM role ARN on the cloudWatchRoleArn
property on your Account. You must
set the cloudWatchRoleArn property separately for each AWS Region in which you
want to enable CloudWatch Logs.
If you receive an error when setting the IAM role ARN, check your AWS Security Token Service account settings to make sure that AWS STS is enabled in the Region that you're using. For more information about enabling AWS STS, see Managing AWS STS in an AWS Region in the IAM User Guide.
Set up CloudWatch API logging using the API Gateway console
To set up CloudWatch API logging, you must have deployed the API to a stage. You must also have configured an appropriate CloudWatch Logs role ARN for your account.
Sign in to the API Gateway console at https://console.aws.amazon.com/apigateway
. Choose a REST API.
-
Choose Settings from the primary navigation panel and enter an ARN of an IAM role with appropriate permissions in CloudWatch log role ARN. You need to do this once.
-
Do one of the following:
-
Choose an existing API and then choose a stage.
-
Create an API and deploy it to a stage.
-
-
Choose Logs/Tracing in the Stage Editor.
-
To enable execution logging:
-
Choose a logging level from the CloudWatch Logs dropdown menu.
Warning
Full Request and Response Logs can be useful to troubleshoot APIs, but can result in logging sensitive data. We recommend that you don't use Full Request and Response Logs for production APIs.
-
If desired, choose Enable Detailed CloudWatch Metrics.
For more information about CloudWatch metrics, see Monitoring REST API execution with Amazon CloudWatch metrics.
-
-
To enable access logging:
-
Choose Enable Access Logging under Custom Access Logging.
-
Enter the ARN of a log group in Access Log Destination ARN. The ARN format is
arn:aws:logs:
.{region}
:{account-id}
:log-group:log-group-name
-
Enter a log format in Log Format. You can choose CLF, JSON, XML, or CSV to use one of the provided examples as a guide.
-
-
Choose Save Changes.
Note
You can enable execution logging and access logging independent of each other.
API Gateway is now ready to log requests to your API. You don't need to redeploy the API when you update the stage settings, logs, or stage variables.
Set up CloudWatch API logging using AWS CloudFormation
Use the following example AWS CloudFormation template to create an Amazon CloudWatch Logs log group and configure execution and access logging for a stage.
TestStage: Type: AWS::ApiGateway::Stage Properties: StageName: test RestApiId: !Ref MyAPI DeploymentId: !Ref Deployment Description: "test stage description" MethodSettings: - ResourcePath: "/*" HttpMethod: "*" LoggingLevel: INFO AccessLogSetting: DestinationArn: !GetAtt MyLogGroup.Arn Format: $context.extendedRequestId $context.identity.sourceIp $context.identity.caller $context.identity.user [$context.requestTime] "$context.httpMethod $context.resourcePath $context.protocol" $context.status $context.responseLength $context.requestId MyLogGroup: Type: AWS::Logs::LogGroup Properties: LogGroupName: !Join - '-' - - !Ref MyAPI - access-logs