Menu
Amazon CloudWatch Events
User Guide

Troubleshooting CloudWatch Events

You can use the steps in this section to troubleshoot CloudWatch Events.

My rule was triggered but my Lambda function was not invoked

Make sure you have the right permissions set for your Lambda function. Run the following command using AWS CLI (replace the function name with your function and use the AWS region your function is in):

aws lambda get-policy --function-name MyFunction --region us-east-1

You should see an output similar to the following:

{
    "Policy": "{\"Version\":\"2012-10-17\",
    \"Statement\":[
        {\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:events:us-east-1:123456789012:rule/MyRule\"}},
        \"Action\":\"lambda:InvokeFunction\",
        \"Resource\":\"arn:aws:lambda:us-east-1:123456789012:function:MyFunction\",
        \"Effect\":\"Allow\",
        \"Principal\":{\"Service\":\"events.amazonaws.com\"},
        \"Sid\":\"MyId\"}
    ],
    \"Id\":\"default\"}"
}

If you see the following:

A client error (ResourceNotFoundException) occurred when calling the GetPolicy operation: The resource you requested does not exist.

Or, you see the output but you can't locate events.amazonaws.com as a trusted entity in the policy, run the following command:

aws lambda add-permission \
--function-name MyFunction \
--statement-id MyId \
--action 'lambda:InvokeFunction' \
--principal events.amazonaws.com \
--source-arn arn:aws:events:us-east-1:123456789012:rule/MyRule

Note

If the policy is incorrect, you can also edit the rule in the CloudWatch Events console by removing and then adding it back to the rule. The CloudWatch Events console will set the correct permissions on the target.

If you're using a specific Lambda alias or version, you must add the --qualifier parameter in the aws lambda get-policy and aws lambda add-permission commands.

aws lambda add-permission \
--function-name MyFunction \
--statement-id MyId \
--action 'lambda:InvokeFunction' \
--principal events.amazonaws.com \
--source-arn arn:aws:events:us-east-1:123456789012:rule/MyRule
--qualifier alias or version

I have just created/modified a rule but it did not match a test event

When you make a change to a rule or to its targets, incoming events might not immediately start or stop matching to new or updated rules. Please allow a short period of time for changes to take effect. If, after this short period, events still do not match, you can also check several Events metrics for your rule in CloudWatch such as TriggeredRules, Invocations, and FailedInvocations for further debugging.

You can also use the TestEventPattern action to test the event pattern of your rule with a test event to make sure the event pattern of your rule is correctly set. For more information, see TestEventPattern in the Amazon CloudWatch Events API Reference.

My rule did not self-trigger at the time specified in the ScheduleExpression

ScheduleExpressions are in UTC. Make sure you have set the schedule for rule to self-trigger in the UTC timezone. If the ScheduleExpression is correct, then follow the steps under I have just created/modified a rule but it did not match a test event.

My rule did not trigger at the time that I expected

CloudWatch Events doesn't support setting an exact start time when you create a rule to run every time period. The count down to run time begins as soon as you create the rule.

You can use a cron expression to invoke targets at a specified time. For example, you can use a cron expression to create a rule that is triggered every 4 hours exactly on 0 minute. In the CloudWatch console, you'd use the cron expression 0 0/4 * * ? *, and with the AWS CLI you'd use the cron expression cron(0 0/4 * * ? *). For example, to create a rule named TestRule that is triggered every 4 hours using the AWS CLI, you would type the following at a command prompt:

aws events put-rule --name TestRule --schedule-expression 'cron(0 0/4 * * ? *)'

You can use the 0/5 * * * ? * cron expression to trigger a rule every 5 minutes. For example:

aws events put-rule --name TestRule --schedule-expression 'cron(0/5 * * * ? *)'

CloudWatch Events does not provide second-level precision in schedule expressions. The finest resolution using a cron expression is a minute. Due to the distributed nature of the CloudWatch Events and the target services, the delay between the time the scheduled rule is triggered and the time the target service honors the execution of the target resource might be several seconds. Your scheduled rule will be triggered within that minute but not on the precise 0th second.

My rule matches IAM API calls but my rule was not triggered

The IAM service is only available in the US East (N. Virginia) Region, so any AWS API call events from IAM are only available in that region. For more information, see Event Types for CloudWatch Events.

My rule is not working because the IAM role associated with the rule is ignored when the rule is triggered

IAM roles for rules are only used for relating events to Amazon Kinesis streams. For Lambda functions and Amazon SNS topics, you need to provide resource-based permissions.

Make sure your regional AWS STS endpoints are enabled. CloudWatch Events talks to the regional AWS STS endpoints when assuming the IAM role you provided. For more information, see Activating and Deactivating AWS STS in an AWS Region in the IAM User Guide.

I created a rule with an EventPattern that is supposed to match a resource, but I don't see any events that match the rule

Most services in AWS treat : or / as the same character in Amazon Resource Names (ARNs). However, CloudWatch Events uses an exact match in event patterns and rules. Be sure to use the correct ARN characters when creating event patterns so that they match the ARN syntax in the event you want to match.

Moreover, not every event has the resources field populated (e.g. AWS API Call events from CloudTrail).

My event's delivery to the target experienced a delay

Amazon CloudWatch Events tries to deliver an event to a target for up to 24 hours. The first attempt is made as soon as the event arrives in the event stream. However, if the target service is having problems or your account is being throttled, CloudWatch Events automatically reschedules another delivery in the future. If 24 hours has passed since the arrival of event, no more attempts are scheduled and the FailedInvocations metric is published in Amazon CloudWatch.

My rule was triggered more than once in response two identical events. What guarantee does CloudWatch Events offer for triggering rules or delivering events to the targets?

Amazon CloudWatch Events guarantees triggering a rule at least once in response to an event. In rare cases, the same rule can be triggered more than once for a given event, or the same target can be invoked more than once for a given triggered rule.

My rule is being triggered but I don't see any messages published into my Amazon SNS topic

Make sure you have the right permission set for your Amazon SNS topic. Run the following command using AWS CLI (replace the topic ARN with your topic and use the AWS region your topic is in):

aws sns get-topic-attributes --region us-east-1 --topic-arn "arn:aws:sns:us-east-1:123456789012:MyTopic"

You should see policy attribute similar to the following:

"{\"Version\":\"2012-10-17\",
\"Id\":\"__default_policy_ID\",
\"Statement\":[{\"Sid\":\"__default_statement_ID\",
\"Effect\":\"Allow\",
\"Principal\":{\"AWS\":\"*\"},
\"Action\":[\"SNS:Subscribe\",
\"SNS:ListSubscriptionsByTopic\",
\"SNS:DeleteTopic\",
\"SNS:GetTopicAttributes\",
\"SNS:Publish\",
\"SNS:RemovePermission\",
\"SNS:AddPermission\",
\"SNS:Receive\",
\"SNS:SetTopicAttributes\"],
\"Resource\":\"arn:aws:sns:us-east-1:123456789012:MyTopic\",
\"Condition\":{\"StringEquals\":{\"AWS:SourceOwner\":\"123456789012\"}}},{\"Sid\":\"Allow_Publish_Events\",
\"Effect\":\"Allow\",
\"Principal\":{\"Service\":\"events.amazonaws.com\"},
\"Action\":\"sns:Publish\",
\"Resource\":\"arn:aws:sns:us-east-1:123456789012:MyTopic\"}]}"

If you see a policy similar to the following, you have only the default policy set:

"{\"Version\":\"2008-10-17\",
\"Id\":\"__default_policy_ID\",
\"Statement\":[{\"Sid\":\"__default_statement_ID\",
\"Effect\":\"Allow\",
\"Principal\":{\"AWS\":\"*\"},
\"Action\":[\"SNS:Subscribe\",
\"SNS:ListSubscriptionsByTopic\",
\"SNS:DeleteTopic\",
\"SNS:GetTopicAttributes\",
\"SNS:Publish\",
\"SNS:RemovePermission\",
\"SNS:AddPermission\",
\"SNS:Receive\",
\"SNS:SetTopicAttributes\"],
\"Resource\":\"arn:aws:sns:us-east-1:123456789012:MyTopic\",
\"Condition\":{\"StringEquals\":{\"AWS:SourceOwner\":\"123456789012\"}}}]}"

If you don't see events.amazonaws.com with Publish permission in your policy, use AWS CLI to set topic policy attribute.

Copy current policy and add statement below to list of statements:

{\"Sid\":\"Allow_Publish_Events\",
\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"events.amazonaws.com\"},
\"Action\":\"sns:Publish\",
\"Resource\":\"arn:aws:sns:us-east-1:123456789012:MyTopic\"}

The new policy should look like the one described above.

Set topic attributes with the AWS CLI:

aws sns set-topic-attributes --region us-east-1 --topic-arn "arn:aws:sns:us-east-1:123456789012:MyTopic" --attribute-name Policy --attribute-value NEW_POLICY_STRING

Note

If the policy is incorrect, you can also edit the rule in the CloudWatch Events console by removing and then adding it back to the rule. The CloudWatch Events console will set the correct permissions on the target.

My Amazon SNS topic still has permissions for CloudWatch Events even after I deleted the rule associated with the Amazon SNS topic

When you create a rule with Amazon SNS as the target, CloudWatch Events adds the permission to your Amazon SNS topic on your behalf. If you delete the rule shortly after you create it, CloudWatch Events might be unable to remove the permission from your Amazon SNS topic. If this happens, you can remove the permission from the topic using the aws sns set-topic-attributes command. For more information about resource-based permissions for sending events, see Using Resource-Based Policies for CloudWatch Events.

Which IAM condition keys can I use with CloudWatch Events

Amazon CloudWatch Events supports the AWS-wide condition keys (see Available Keys in the IAM User Guide), plus the following service-specific condition keys. For more information, see Using IAM Policy Conditions for Fine-Grained Access Control.

How can I tell when CloudWatch Events rules are broken

You can use the following alarm to notify you when your CloudWatch Events rules are broken.

To create an alarm to alert when rules are broken

  1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.

  2. Click Create Alarm, and then in the CloudWatch Metrics by Category pane, select Events Metrics.

  3. On the Create Alarm dialog box, in the list of metrics, select the FailedInvocations check box.

  4. Above the graph, select Sum from the Statistic drop-down list.

  5. Select a period from the Period drop-down list, for example: 5 minutes.

  6. Click Next, and then under Alarm Threshold, in the Name field, enter a unique name for the alarm, for example: myFailedRules.

  7. In the Description field, enter a description of the alarm, for example: Rules are not delivering events to targets.

  8. In the is drop-down list, select >=.

  9. In the field next to the is drop-down list, enter 1 and in the for field, enter 10.

  10. Under Actions, in the Whenever this alarm drop-down list, select State is ALARM.

  11. In the Send notification to drop-down list, select an existing Amazon SNS topic or create a new one.

  12. To create a new Amazon SNS topic, select New list.

  13. In the Send notification to field, enter a name for the new Amazon SNS topic for example: myFailedRules, and in the Email list field, enter a comma-separated list of email addresses to be notified when the alarm changes to the ALARM state.

  14. In the navigation pane, choose Create Alarm to complete the alarm creation process.