Amazon MSK cluster target
To deliver to an Amazon MSK cluster, set TargetArn to the cluster ARN and add
MskParameters with the topic to produce to. EventBridge produces one Kafka record per delivered
event: the event is the record value. MskParameters sets each record's topic, key, and headers,
and the producer's compression and acknowledgement mode. The cluster must have the connectivity path you name turned on, with
IAM access control. For more information about connectivity, see Public and private connectivity.
Parameters
The following table describes the MskParameters fields.
| Parameter | What it sets |
|---|---|
TopicName | Required. The Kafka topic EventBridge produces to. A literal, or a JSONata expression resolved once per delivered event. Every resolved value must be a valid Kafka topic name. |
Connectivity | Required. PUBLIC to reach the cluster over its public access, or PRIVATE over a private connection EventBridge establishes. A provisioned cluster can have both turned on, so you name the one to use. An Amazon MSK Serverless cluster supports only PRIVATE. See Public and private connectivity. |
PartitionKey | The Kafka record key. A literal, or a JSONata expression resolved once per delivered event; the resolved value selects the partition. If you omit it, a FIFO subscriber uses the event's event group ID as the record key, so events in a group share a partition; a standard subscriber lets the producer choose. |
Compression | The producer compression codec applied to the record batch: NONE (the default), GZIP, SNAPPY, LZ4, or ZSTD. |
Acks | The producer acknowledgement mode. ALL (the default) waits for all in-sync replicas; LEADER waits for the partition leader only. |
Headers | Kafka record headers attached to each record, up to 100. Each header has a key and an optional value, each a literal or a JSONata expression resolved once per delivered event. |
ClientId | The producer client.id sent to the broker, which identifies the producer in broker-side logs, metrics, and quotas. A literal, set once when the producer connects. |
Public and private connectivity
EventBridge reaches the cluster over the path you name in Connectivity. Before you create the
subscriber, turn on IAM access control on the cluster, and set up the path you name:
-
For
PUBLIC, turn on the cluster's public access. This option applies to provisioned clusters only. -
For
PRIVATEon a provisioned cluster, turn on the cluster's multi-VPC private connectivity with IAM, and add a cluster policy that allows EventBridge to connect. -
For
PRIVATEon an Amazon MSK Serverless cluster, add a cluster policy that allows EventBridge to connect. Serverless clusters have no multi-VPC setting to turn on.
In both PRIVATE cases, EventBridge establishes the private connection to your cluster.
Private connectivity creates an AWS PrivateLink VPC endpoint to connect to the cluster, billed to the AWS
account that owns the cluster. Subscribers that target the same cluster share one endpoint. You do not create or manage any networking resources for it.
Amazon MSK Serverless clusters have no public access, so CreateSubscriber and
UpdateSubscriber reject PUBLIC connectivity for them. Private delivery to a
Serverless cluster is available only in Regions where Amazon MSK Serverless is available; in other Regions,
the subscriber's ConnectivityState becomes PROVISION_FAILED.
For the PRIVATE path, attach a cluster policy that lets EventBridge set up and use the private
connection to your cluster, scoped with aws:SourceAccount and aws:SourceArn to the subscribers that deliver to it.
The following example grants that access. In the example, replace the account ID, cluster name, and subscriber name with your own values.
{ "Version": "2012-10-17", "Statement": [ { "Sid": "AllowEventBridgePrivateConnectivity", "Effect": "Allow", "Principal": { "Service": "events.amazonaws.com" }, "Action": [ "kafka:CreateVpcConnection", "kafka:GetBootstrapBrokers", "kafka:DescribeClusterV2" ], "Resource": "arn:aws:kafka:us-east-1:111122223333:cluster/MyCluster/abcd1234-0abc-def0-1234-abcd1234abcd-25", "Condition": { "StringEquals": { "aws:SourceAccount": "111122223333" }, "ArnLike": { "aws:SourceArn": "arn:aws:events:us-east-1:111122223333:subscriber/my-subscriber/*" } } } ] }
The condition uses ArnLike with a trailing wildcard because the subscriber ARN that EventBridge
sends carries a generated ID you cannot know before CreateSubscriber. Each
PRIVATE subscriber's creation is authorized against the cluster policy with its own
ARN, even when it joins an existing shared connection. The condition must therefore match every
PRIVATE subscriber that targets the cluster. Use subscriber/* to match all of
your subscribers, or, for each subscriber, add a subscriber/ pattern, replacing subscriber-name/*subscriber-name with the name of your subscriber.
Provisioning and delivery lifecycle
An Amazon MSK subscriber has two states that you read with DescribeSubscriber. State
is RUNNING or STOPPED and you control it. SubscriberDetails.MskDetails.ConnectivityState
is PROVISIONING, ACTIVE, or PROVISION_FAILED and EventBridge manages it: it
tracks the connectivity EventBridge sets up to the cluster. SubscriberDetails.MskDetails.StateReason explains a failure and is present only when ConnectivityState is PROVISION_FAILED.
-
While
PROVISIONING– after you create aPRIVATEsubscriber, it does not match or deliver events while EventBridge establishes the private connection for the first time. WhenConnectivityStatefirst becomesACTIVE, it begins matching and delivering. A subscriber that has not yet beenACTIVEretains no events, unlike one that re-provisions after having beenACTIVE(described following). APUBLICsubscriber has no connection to establish and becomesACTIVEwithout a provisioning step. -
If the cluster or connection breaks while the subscriber is delivering (for example, you change the cluster so EventBridge can no longer reach it) EventBridge keeps matching events and attempting delivery. The attempts fail and are retried; events that exhaust the retry policy go to the dead-letter queue, or are dropped if none is configured. For more information about dead-letter queues, see Subscribing to events on a Custom Event Bus.
-
To repair a
PRIVATEsubscriber's connectivity, fix the cluster and then callUpdateSubscriberwith the subscriber'sInvokeConfiguration(the sameMskParameters). APUBLICsubscriber has no connection to re-establish, and updating it with the sameMskParametersdoes not re-provision. An update that omits the invoke configuration does not re-establish the connection. You can update a subscriber only fromACTIVEorPROVISION_FAILED; an update while it is alreadyPROVISIONINGis rejected and can be retried.ConnectivityStatereturns toPROVISIONINGwhile EventBridge re-establishes the connection. Because the subscriber has already beenACTIVE, it keeps matching and attempting delivery throughout, so deliveries continue to fail until the connection is restored. When it reachesACTIVE, delivery of the retained events resumes. If provisioning fails,ConnectivityStatebecomesPROVISION_FAILED: EventBridge pauses delivery and keeps matching, so matched events are retained for delivery once you repair the cluster and update the subscriber again.
Subscribers that share a cluster connection
Private subscribers that deliver to the same cluster ARN share one connection to that cluster, so a connection problem affects them together. A shared connection behaves as follows:
-
If the connection breaks, every subscriber that uses it has the same delivery failures, retries, and dead-letter or drop behavior.
-
After you fix the cluster, call
UpdateSubscriberwith itsInvokeConfigurationon any one of those subscribers. When that update rebuilds the connection, EventBridge also returns the otherPROVISION_FAILEDsubscribers on that cluster toACTIVE, provided the cluster policy still allows each one; subscribers alreadyACTIVEstayACTIVE. -
If an update fails because the cluster or its connection is unusable, EventBridge sets this subscriber and the other
ACTIVEsubscribers on that cluster toPROVISION_FAILED. Delivery pauses for those subscribers, but EventBridge keeps matching and retaining their events. A failure caused by one subscriber alone (for example its ownaws:SourceAccountoraws:SourceArncondition) does not move the others.
Delivery role
EventBridge assumes the subscriber's delivery role to produce to your cluster, so the role needs IAM access control permissions to look up the cluster's bootstrap brokers, connect to the cluster, and write to the topic, and must trust EventBridge to assume it (see Subscribing to events on a Custom Event Bus). The following example grants that access for one topic. In the example, replace the account ID, cluster name, and topic name with your own values.
{ "Version": "2012-10-17", "Statement": [ { "Sid": "LookupBrokers", "Effect": "Allow", "Action": "kafka:GetBootstrapBrokers", "Resource": "arn:aws:kafka:us-east-1:111122223333:cluster/MyCluster/abcd1234-0abc-def0-1234-abcd1234abcd-25" }, { "Sid": "Connect", "Effect": "Allow", "Action": "kafka-cluster:Connect", "Resource": "arn:aws:kafka:us-east-1:111122223333:cluster/MyCluster/abcd1234-0abc-def0-1234-abcd1234abcd-25" }, { "Sid": "Produce", "Effect": "Allow", "Action": [ "kafka-cluster:DescribeTopic", "kafka-cluster:WriteData" ], "Resource": "arn:aws:kafka:us-east-1:111122223333:topic/MyCluster/abcd1234-0abc-def0-1234-abcd1234abcd-25/my-topic" } ] }
Batching and ordering
EventBridge produces one Kafka record per event. BatchConfiguration bounds how many records
EventBridge puts into one delivery attempt; the Kafka producer then groups those records into per-partition
broker requests, so one delivery attempt is not a single network call. On a FIFO subscriber, EventBridge
produces an event group's records in the order they were published; set PartitionKey to
keep related events on the same partition. For more information about batching, see Batching deliveries to a target.