Troubleshooting Amazon EventBridge
You can use the topics in this section to troubleshoot Amazon EventBridge issues.
Topics
- My rule ran but my Lambda function wasn't invoked
- I just created or modified a rule, but it didn't match a test event
- My rule didn't run at the time I specified in the ScheduleExpression
- My rule didn't run at the time that I expected
- My rule matches AWS global service API calls but it didn't run
- The IAM role associated with my rule is being ignored when the rule runs
- My rule has an event pattern that is supposed to match a resource, but no events match
- My event's delivery to the target was delayed
- Some events were never delivered to my target
- My rule ran more than once in response to one event
- Preventing infinite loops
- My events are not delivered to the target Amazon SQS queue
- My rule runs, but I don't see any messages published into my Amazon SNS topic
- My Amazon SNS topic still has permissions for EventBridge even after I deleted the rule associated with the Amazon SNS topic
- Which IAM condition keys can I use with EventBridge?
- How can I tell when EventBridge rules are broken?
My rule ran but my Lambda function wasn't invoked
One reason your Lambda function might not run is if you don't have the right permissions.
To check your permissions for your Lambda function
-
Using the AWS CLI, run the following command with your function and your AWS Region:
aws lambda get-policy --function-name
MyFunction
--regionus-east-1
You should see the following output.
{ "Policy": "{\"Version\":\"2012-10-17\", \"Statement\":[ {\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:events:us-east-1:123456789012:rule/MyRule\"}}, \"Action\":\"lambda:InvokeFunction\", \"Resource\":\"arn:aws:lambda:us-east-1:123456789012:function:MyFunction\", \"Effect\":\"Allow\", \"Principal\":{\"Service\":\"events.amazonaws.com\"}, \"Sid\":\"MyId\"} ], \"Id\":\"default\"}" }
-
If you see the following message.
A client error (ResourceNotFoundException) occurred when calling the GetPolicy operation: The resource you requested does not exist.
Or, you see the output but you can't locate events.amazonaws.com as a trusted entity in the policy, run the following command:
aws lambda add-permission \ --function-name
MyFunction
\ --statement-idMyId
\ --action 'lambda:InvokeFunction' \ --principal events.amazonaws.com \ --source-arn arn:aws:events:us-east-1:123456789012
:rule/MyRule
-
If the output contains a
SourceAccount
field, then you need to remove it. ASourceAccount
setting prevents EventBridge from being able to invoke the function.
Note
If the policy is incorrect, you can edit the rule in the EventBridge console by removing and then adding it back to the rule. The EventBridge console then sets the correct permissions on the target.
If you're using a specific Lambda alias or version, add the
--qualifier
parameter in the aws lambda get-policy
and
aws lambda add-permission
commands, as shown in the following command
aws lambda add-permission \ --function-name
MyFunction
\ --statement-idMyId
\ --action 'lambda:InvokeFunction' \ --principal events.amazonaws.com \ --source-arn arn:aws:events:us-east-1:123456789012
:rule/MyRule
--qualifieralias or version
I just created or modified a rule, but it didn't match a test event
When you make a change to a rule or to its targets, incoming events might not immediately start or stop matching to new or updated rules. Allow a short period of time for changes to take effect.
If events still don't match after a short period of time, check the CloudWatch metrics
TriggeredRules
, Invocations
, and
FailedInvocations
for your rule. For more information about these
metrics, see Monitoring Amazon EventBridge.
If the rule is intended to match an event from an AWS service, do one of these things:
Use the
TestEventPattern
action to test the event pattern of your rule matches a test event. For more information, see TestEventPattern in the Amazon EventBridge API Reference.Use the Sandbox on the EventBridge console
.
My rule didn't run at the time I specified in
the ScheduleExpression
Make sure you have set the schedule for the rule in
the UTC+0 time zone. If the ScheduleExpression
is correct, then follow the
steps under I just created or modified a rule, but it
didn't match a test event.
My rule didn't run at the time that I expected
EventBridge runs rules within one minute of the start time you set. The count down to run time begins as soon as you create the rule.
Note
Scheduled rules have delivery type of guaranteed
meaning events will be triggered for each expected time at least once.
You can use a cron expression to invoke targets at a specified time. To create a rule that runs every four hours on the 0th minute, you do one of the following:
-
In the EventBridge console, you use the cron expression
0 0/4 * * ? *
. -
Using the AWS CLI, you use the expression
cron(0 0/4 * * ? *)
.
For example, to create a rule named TestRule
that runs every 4 hours by
using the AWS CLI, you use the following command.
aws events put-rule --name
TestRule
--schedule-expression 'cron(0 0/4 * * ? *)'
To run a rule every five minutes, you use the following cron expressio.
aws events put-rule --name
TestRule
--schedule-expression 'cron(0/5 * * * ? *)'
The finest resolution for an EventBridge rule that uses a cron expression is one minute. Your scheduled rule runs within that minute but not on the precise 0th second.
Because EventBridge and target services are distributed, there can be a delay of several seconds between the time the scheduled rule runs and the time the target service performs the action on the target resource.
My rule matches AWS global service API calls but it didn't run
AWS global services; such as, IAM and Amazon RouteĀ 53 are only available in the US East (N. Virginia) Region, so events from AWS API calls from global services are only available in that region. For more information, see Events from AWS services in Amazon EventBridge.
The IAM role associated with my rule is being ignored when the rule runs
EventBridge only uses IAM roles for rules that send events to Kinesis streams. For rules that invoke Lambda functions or Amazon SNS topics, you need to provide resource-based permissions.
Make sure your regional AWS STS endpoints are enabled, so that EventBridge can use them when assuming the IAM role you provided. For more information, see Activating and Deactivating AWS STS in an AWS Region in the IAM User Guide.
My rule has an event pattern that is supposed to match a resource, but no events match
Most services in AWS treat a colon (:) or slash (/) as the same character in Amazon Resource Names (ARNs)., but EventBridge uses an exact match in event patterns and rules. Be sure to use the correct ARN characters when creating event patterns so that they match the ARN syntax in the event to match.
Some events, such as AWS API call events from CloudTrail, don't have anything in the resources field.
My event's delivery to the target was delayed
EventBridge tries to deliver an event to a target for up to 24 hours, except in scenarios where
your target resource is constrained. The first attempt is made as soon as the event
arrives in the event stream. If the target service is having problems, EventBridge
automatically reschedules another delivery. If 24 hours has passed since the arrival of
event, EventBridge stops trying to deliver the event and publishes the
FailedInvocations
metric in CloudWatch. We recommend that you set up a DLQ to store events that couldn't
successfully be delivered to a target. For more information, see Using dead-letter queues to process undelivered events in EventBridge
Some events were never delivered to my target
If the target of an EventBridge rule is constrained for a prolonged time, EventBridge might not retry delivery. For example, if the target is not provisioned to handle the incoming event traffic and the target service is throttling requests that EventBridge makes on your behalf, then EventBridge might not retry delivery.
My rule ran more than once in response to one event
In rare cases, the same rule can run more than once for a single event or scheduled time, or the same target can be invoked more than once for a given triggered rule.
Preventing infinite loops
In EventBridge, it is possible to create a rule that leads to infinite loops, where the rule runs repeatedly. If you have a rule that causes an infinite loop, rewrite it so that the actions that the rule takes don't match the same rule.
For example, a rule that detects that ACLs have changed on an Amazon S3 bucket and then runs software to change them to a new state causes an infinite loop. One way to resolve it is to rewrite the rule so that it only matches ACLs that are in a bad state.
An infinite loop can quickly cause higher than expected charges. We recommend that you use budgeting, which alerts you when charges exceed your specified limit. For more information, see Managing Your Costs with Budgets.
My events are not delivered to the target Amazon SQS queue
If your Amazon SQS queue is encrypted, you must create a customer-managed KMS key and include the following permission section in your KMS key policy. For more information, see Configuring AWS KMS permissions.
{ "Sid": "Allow EventBridge to use the key", "Effect": "Allow", "Principal": { "Service": "events.amazonaws.com" }, "Action": [ "kms:Decrypt", "kms:GenerateDataKey" ], "Resource": "*" }
My rule runs, but I don't see any messages published into my Amazon SNS topic
Scenario 1
You need permission for messages to be published into your Amazon SNS topic. Use the following command using the AWS CLI, replacing us-east-1 with your Region and using your topic ARN.
aws sns get-topic-attributes --region
us-east-1
--topic-arn "arn:aws:sns:us-east-1
:123456789012
:MyTopic"
To have the correct permission, your policy attributes similar to the following.
"{\"Version\":\"2012-10-17\", \"Id\":\"__default_policy_ID\", \"Statement\":[{\"Sid\":\"__default_statement_ID\", \"Effect\":\"Allow\", \"Principal\":{\"AWS\":\"*\"}, \"Action\":[\"SNS:Subscribe\", \"SNS:ListSubscriptionsByTopic\", \"SNS:DeleteTopic\", \"SNS:GetTopicAttributes\", \"SNS:Publish\", \"SNS:RemovePermission\", \"SNS:AddPermission\", \"SNS:SetTopicAttributes\"], \"Resource\":\"arn:aws:sns:us-east-1:123456789012:MyTopic\", \"Condition\":{\"StringEquals\":{\"AWS:SourceOwner\":\"123456789012\"}}},{\"Sid\":\"Allow_Publish_Events\", \"Effect\":\"Allow\", \"Principal\":{\"Service\":\"events.amazonaws.com\"}, \"Action\":\"sns:Publish\", \"Resource\":\"arn:aws:sns:us-east-1:123456789012:MyTopic\"}]}"
If you don't see events.amazonaws.com
with Publish
permission in your policy, first copy the current policy and add the following
statement to the list of statements.
{\"Sid\":\"Allow_Publish_Events\", \"Effect\":\"Allow\",\"Principal\":{\"Service\":\"events.amazonaws.com\"}, \"Action\":\"sns:Publish\", \"Resource\":\"arn:aws:sns:us-east-1:123456789012:MyTopic\"}
Then set the topic attributes by using the AWS CLI, use the following command.
aws sns set-topic-attributes --region
us-east-1
--topic-arn "arn:aws:sns:us-east-1:123456789012
:MyTopic" --attribute-name Policy --attribute-valueNEW_POLICY_STRING
Note
Scenario 2
If your SNS topic is encrypted, you must include the following section in your KMS key policy.
{ "Sid": "Allow EventBridge to use the key", "Effect": "Allow", "Principal": { "Service": "events.amazonaws.com" }, "Action": [ "kms:Decrypt", "kms:GenerateDataKey" ], "Resource": "*" }
My Amazon SNS topic still has permissions for EventBridge even after I deleted the rule associated with the Amazon SNS topic
When you create a rule with Amazon SNS as the target, EventBridge adds permission to your Amazon SNS topic on your
behalf. If you delete the rule shortly after you create it, EventBridge might not remove the
permission from your Amazon SNS topic. If this happens, you can remove the permission from
the topic by using the aws sns set-topic-attributes
command. For information about
resource-based permissions for sending events, see Using resource-based policies for Amazon EventBridge.
Which IAM condition keys can I use with EventBridge?
EventBridge supports the AWS-wide condition keys (see IAM and AWS STS condition context keys in the IAM User Guide), plus the keys listed at Using IAM policy conditions in Amazon EventBridge.
How can I tell when EventBridge rules are broken?
You can use the following alarm to notify you when your EventBridge rules are broken.
To create an alarm to alert when rules are broken
-
Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/
. -
Choose Create Alarm. In the CloudWatch Metrics by Category pane, choose Events Metrics.
-
In the list of metrics, select FailedInvocations.
-
Above the graph, choose Statistic, Sum.
-
For Period, choose a value, for example 5 minutes. Choose Next.
-
Under Alarm Threshold, for Name, type a unique name for the alarm, for example myFailedRules. For Description, type a description of the alarm, for example Rules aren't delivering events to targets.
-
For is, choose >= and 1. For for, enter 10.
-
Under Actions, for Whenever this alarm, choose State is ALARM.
-
For Send notification to, select an existing Amazon SNS topic or create a new one. To create a new topic, choose New list. Type a name for the new Amazon SNS topic, for example: myFailedRules.
-
For Email list, type a comma-separated list of email addresses to be notified when the alarm changes to the ALARM state.
-
Choose Create Alarm.