Troubleshooting Amazon EventBridge - Amazon EventBridge

Troubleshooting Amazon EventBridge

You can use the steps in this section to troubleshoot Amazon EventBridge.

My rule ran but my Lambda function wasn't invoked

One reason your Lambda function might not run is if you don't have the right permissions.

To check your permissions for your Lambda function
  1. Using the AWS CLI, run the following command with your function and your AWS Region:

    aws lambda get-policy --function-name MyFunction --region us-east-1

    You should see the following output.

    { "Policy": "{\"Version\":\"2012-10-17\", \"Statement\":[ {\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:events:us-east-1:123456789012:rule/MyRule\"}}, \"Action\":\"lambda:InvokeFunction\", \"Resource\":\"arn:aws:lambda:us-east-1:123456789012:function:MyFunction\", \"Effect\":\"Allow\", \"Principal\":{\"Service\":\"events.amazonaws.com\"}, \"Sid\":\"MyId\"} ], \"Id\":\"default\"}" }
  2. If you see the following message.

    A client error (ResourceNotFoundException) occurred when calling the GetPolicy operation: The resource you requested does not exist.

    Or, you see the output but you can't locate events.amazonaws.com as a trusted entity in the policy, run the following command:

    aws lambda add-permission \ --function-name MyFunction \ --statement-id MyId \ --action 'lambda:InvokeFunction' \ --principal events.amazonaws.com \ --source-arn arn:aws:events:us-east-1:123456789012:rule/MyRule
  3. If the output contains a SourceAccount field, then you need to remove it. A SourceAccount setting prevents EventBridge from being able to invoke the function.

Note

If the policy is incorrect, you can edit the rule in the EventBridge console by removing and then adding it back to the rule. The EventBridge console then sets the correct permissions on the target.

If you're using a specific Lambda alias or version, add the --qualifier parameter in the aws lambda get-policy and aws lambda add-permission commands, as shown in the following command

aws lambda add-permission \ --function-name MyFunction \ --statement-id MyId \ --action 'lambda:InvokeFunction' \ --principal events.amazonaws.com \ --source-arn arn:aws:events:us-east-1:123456789012:rule/MyRule --qualifier alias or version

I just created or modified a rule, but it didn't match a test event

When you make a change to a rule or to its targets, incoming events might not immediately start or stop matching to new or updated rules. Allow a short period of time for changes to take effect.

If events still don't match after a short period of time, check the CloudWatch metrics TriggeredRules, Invocations, and FailedInvocations for your rule. For more information about these metrics, see Monitoring Amazon EventBridge.

If the rule is intended to match an event from an AWS service, do one of these things:

  • Use the TestEventPattern action to test the event pattern of your rule matches a test event. For more information, see TestEventPattern in the Amazon EventBridge API Reference.

  • Use the Sandbox on the EventBridge console.

My rule didn't run at the time I specified in the ScheduleExpression

Make sure you have set the schedule for the rule in the UTC+0 time zone. If the ScheduleExpression is correct, then follow the steps under I just created or modified a rule, but it didn't match a test event.

My rule didn't run at the time that I expected

EventBridge runs rules within one minute of the start time you set. The count down to run time begins as soon as you create the rule.

Note

Scheduled rules have delivery type of guaranteed meaning events will be triggered for each expected time at least once.

You can use a cron expression to invoke targets at a specified time. To create a rule that runs every four hours on the 0th minute, you do one of the following:

  • In the EventBridge console, you use the cron expression 0 0/4 * * ? *.

  • Using the AWS CLI, you use the expression cron(0 0/4 * * ? *).

For example, to create a rule named TestRule that runs every 4 hours by using the AWS CLI, you use the following command.

aws events put-rule --name TestRule --schedule-expression 'cron(0 0/4 * * ? *)'

To run a rule every five minutes, you use the following cron expressio.

aws events put-rule --name TestRule --schedule-expression 'cron(0/5 * * * ? *)'

The finest resolution for an EventBridge rule that uses a cron expression is one minute. Your scheduled rule runs within that minute but not on the precise 0th second.

Because EventBridge and target services are distributed, there can be a delay of several seconds between the time the scheduled rule runs and the time the target service performs the action on the target resource.

My rule matches AWS global service API calls but it didn't run

AWS global services; such as, IAM and Amazon RouteĀ 53 are only available in the US East (N. Virginia) Region, so events from AWS API calls from global services are only available in that region. For more information, see Events from AWS services.

The IAM role associated with my rule is being ignored when the rule runs

EventBridge only uses IAM roles for rules that send events to Kinesis streams. For rules that invoke Lambda functions or Amazon SNS topics, you need to provide resource-based permissions.

Make sure your regional AWS STS endpoints are enabled, so that EventBridge can use them when assuming the IAM role you provided. For more information, see Activating and Deactivating AWS STS in an AWS Region in the IAM User Guide.

My rule has an event pattern that is supposed to match a resource, but no events match

Most services in AWS treat a colon (:) or slash (/) as the same character in Amazon Resource Names (ARNs)., but EventBridge uses an exact match in event patterns and rules. Be sure to use the correct ARN characters when creating event patterns so that they match the ARN syntax in the event to match.

Some events, such as AWS API call events from CloudTrail, don't have anything in the resources field.

My event's delivery to the target was delayed

EventBridge tries to deliver an event to a target for up to 24 hours, except in scenarios where your target resource is constrained. The first attempt is made as soon as the event arrives in the event stream. If the target service is having problems, EventBridge automatically reschedules another delivery. If 24 hours has passed since the arrival of event, EventBridge stops trying to deliver the event and publishes the FailedInvocations metric in CloudWatch. We recommend that you set up a DLQ to store events that couldn't successfully be delivered to a target. For more information, see Event retry policy and using dead-letter queues

Some events were never delivered to my target

If the target of an EventBridge rule is constrained for a prolonged time, EventBridge might not retry delivery. For example, if the target is not provisioned to handle the incoming event traffic and the target service is throttling requests that EventBridge makes on your behalf, then EventBridge might not retry delivery.

My rule ran more than once in response to one event

In rare cases, the same rule can run more than once for a single event or scheduled time, or the same target can be invoked more than once for a given triggered rule.

Preventing infinite loops

In EventBridge, it is possible to create a rule that leads to infinite loops, where the rule runs repeatedly. If you have a rule that causes an infinite loop, rewrite it so that the actions that the rule takes don't match the same rule.

For example, a rule that detects that ACLs have changed on an Amazon S3 bucket and then runs software to change them to a new state causes an infinite loop. One way to resolve it is to rewrite the rule so that it only matches ACLs that are in a bad state.

An infinite loop can quickly cause higher than expected charges. We recommend that you use budgeting, which alerts you when charges exceed your specified limit. For more information, see Managing Your Costs with Budgets.

My events are not delivered to the target Amazon SQS queue

If your Amazon SQS queue is encrypted, you must create a customer-managed KMS key and include the following permission section in your KMS key policy. For more information, see Configuring AWS KMS permissions.

{ "Sid": "Allow EventBridge to use the key", "Effect": "Allow", "Principal": { "Service": "events.amazonaws.com" }, "Action": [ "kms:Decrypt", "kms:GenerateDataKey" ], "Resource": "*" }

My rule runs, but I don't see any messages published into my Amazon SNS topic

Scenario 1

You need permission for messages to be published into your Amazon SNS topic. Use the following command using the AWS CLI, replacing us-east-1 with your Region and using your topic ARN.

aws sns get-topic-attributes --region us-east-1 --topic-arn "arn:aws:sns:us-east-1:123456789012:MyTopic"

To have the correct permission, your policy attributes similar to the following.

"{\"Version\":\"2012-10-17\", \"Id\":\"__default_policy_ID\", \"Statement\":[{\"Sid\":\"__default_statement_ID\", \"Effect\":\"Allow\", \"Principal\":{\"AWS\":\"*\"}, \"Action\":[\"SNS:Subscribe\", \"SNS:ListSubscriptionsByTopic\", \"SNS:DeleteTopic\", \"SNS:GetTopicAttributes\", \"SNS:Publish\", \"SNS:RemovePermission\", \"SNS:AddPermission\", \"SNS:SetTopicAttributes\"], \"Resource\":\"arn:aws:sns:us-east-1:123456789012:MyTopic\", \"Condition\":{\"StringEquals\":{\"AWS:SourceOwner\":\"123456789012\"}}},{\"Sid\":\"Allow_Publish_Events\", \"Effect\":\"Allow\", \"Principal\":{\"Service\":\"events.amazonaws.com\"}, \"Action\":\"sns:Publish\", \"Resource\":\"arn:aws:sns:us-east-1:123456789012:MyTopic\"}]}"

If you don't see events.amazonaws.com with Publish permission in your policy, first copy the current policy and add the following statement to the list of statements.

{\"Sid\":\"Allow_Publish_Events\", \"Effect\":\"Allow\",\"Principal\":{\"Service\":\"events.amazonaws.com\"}, \"Action\":\"sns:Publish\", \"Resource\":\"arn:aws:sns:us-east-1:123456789012:MyTopic\"}

Then set the topic attributes by using the AWS CLI, use the following command.

aws sns set-topic-attributes --region us-east-1 --topic-arn "arn:aws:sns:us-east-1:123456789012:MyTopic" --attribute-name Policy --attribute-value NEW_POLICY_STRING
Note

If the policy is incorrect, you can also edit the rule in the EventBridge console by removing and then adding it back to the rule. EventBridge sets the correct permissions on the target.

Scenario 2

If your SNS topic is encrypted, you must include the following section in your KMS key policy.

{ "Sid": "Allow EventBridge to use the key", "Effect": "Allow", "Principal": { "Service": "events.amazonaws.com" }, "Action": [ "kms:Decrypt", "kms:GenerateDataKey" ], "Resource": "*" }

My Amazon SNS topic still has permissions for EventBridge even after I deleted the rule associated with the Amazon SNS topic

When you create a rule with Amazon SNS as the target, EventBridge adds permission to your Amazon SNS topic on your behalf. If you delete the rule shortly after you create it, EventBridge might not remove the permission from your Amazon SNS topic. If this happens, you can remove the permission from the topic by using the aws sns set-topic-attributes command. For information about resource-based permissions for sending events, see Using resource-based policies for Amazon EventBridge.

Which IAM condition keys can I use with EventBridge?

EventBridge supports the AWS-wide condition keys (see IAM and AWS STS condition context keys in the IAM User Guide), plus the keys listed at Using IAM policy conditions for fine-grained access control.

How can I tell when EventBridge rules are broken?

You can use the following alarm to notify you when your EventBridge rules are broken.

To create an alarm to alert when rules are broken
  1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.

  2. Choose Create Alarm. In the CloudWatch Metrics by Category pane, choose Events Metrics.

  3. In the list of metrics, select FailedInvocations.

  4. Above the graph, choose Statistic, Sum.

  5. For Period, choose a value, for example 5 minutes. Choose Next.

  6. Under Alarm Threshold, for Name, type a unique name for the alarm, for example myFailedRules. For Description, type a description of the alarm, for example Rules aren't delivering events to targets.

  7. For is, choose >= and 1. For for, enter 10.

  8. Under Actions, for Whenever this alarm, choose State is ALARM.

  9. For Send notification to, select an existing Amazon SNS topic or create a new one. To create a new topic, choose New list. Type a name for the new Amazon SNS topic, for example: myFailedRules.

  10. For Email list, type a comma-separated list of email addresses to be notified when the alarm changes to the ALARM state.

  11. Choose Create Alarm.