Amazon MSK data delivery to Amazon S3
Amazon MSK data delivery to Amazon S3 delivers your Apache Kafka data to general purpose Amazon S3 buckets in its source format. You choose a Kafka topic and a destination bucket, and Amazon MSK Express brokers deliver your records to Amazon S3 as a fully managed capability.
Amazon MSK scales delivery to your workload automatically and handles retries and backpressure, supporting throughput of up to 10 GBps. Routine operations such as capacity scaling and version upgrades happen without delivery gaps. Because delivery is native to Express brokers rather than a connector fleet that you run, there is no additional broker egress throughput to provision, and you pay $8.00 per TB delivered. Together this can reduce ingestion and delivery costs by up to 60% compared to self-managed alternatives.
Without this capability, delivering Apache Kafka data to Amazon S3 for log archival, compliance retention, Kafka replay, or machine learning training data means building pipelines from self-managed connectors. You source or build connector plugins, secure approvals to deploy them, scale worker capacity as throughput grows, and apply security updates across a connector fleet, and you size that capacity for peak rather than actual demand. Data delivery removes the connector fleet and the coordination it requires.
The following diagram shows how records flow from an Amazon MSK Express broker topic through a Data Delivery channel to a general purpose Amazon S3 bucket, shown as the dashed path. The solid path shows delivery to Apache Iceberg tables in Amazon S3 Tables.
Topics
Benefits
Amazon MSK data delivery to Amazon S3 provides the following benefits:
No infrastructure to manage — No connectors or compute clusters. You configure a Channel and the service handles delivery, scaling, and fault tolerance.
No broker impact — A channel reads from the topic without consuming broker throughput or affecting producer and consumer workloads.
Scales with your data — Supports data delivery throughput of up to 10 GBps with no manual scaling required.
Data freshness in minutes — Delivered data is available for querying or processing within 5 to 15 minutes of being produced to the topic.
Built-in error handling — Unprocessable records are routed to a dead-letter queue with error context, so delivery continues uninterrupted.
Pricing
You pay for the volume of data delivered from your Apache Kafka topics to the destination, billed at per-byte resolution, at $8.00 per TB. There are no setup fees, minimum commitments, or upfront costs.
Standard Amazon S3 storage, request, and data transfer charges apply to the destination bucket. There is no additional charge for broker egress used by this capability, and there are no separate connector, worker, or MSK Connect Unit (MCU) fees.
You are not charged separately for failed delivery attempts routed to the dead-letter queue. Only successfully delivered data is billed.
Rates vary by destination type and are subject to change. For current pricing, see
Amazon MSK pricing
Common use cases
Common use cases for data delivery to Amazon S3 include the following:
Archive Kafka topic data to S3 for storage, replay, or downstream batch processing.
Fan out a single topic to multiple destinations without adding broker load.
For the API specification, see CreateChannel, DescribeChannel, UpdateChannel, DeleteChannel, and ListChannels in the Amazon MSK API Reference.
Integrations
Data delivery to Amazon S3 integrates with the following AWS services:
Amazon MSK Express brokers — the data source.
Amazon S3 — general-purpose object destination.
Amazon CloudWatch — metrics and operational logs.
AWS CloudTrail — API audit logging.
AWS KMS — optional customer-managed encryption at rest.
How it works
To deliver data to a general-purpose S3 bucket, you create a Channel. You create a Channel on an Amazon MSK Provisioned cluster that uses Express brokers. The Channel reads records from a Kafka topic and delivers them to the configured destination.
For Amazon S3 general purpose buckets, the Channel writes records (JSON, ByteArray, or String) as objects to a general-purpose S3 bucket, using a configurable output key template.
Records that cannot be processed are routed to a required dead-letter queue (DLQ) S3 bucket.
Note
A Channel does not backfill previously produced data — only data produced after enablement is delivered.
Requirements and supported configurations
Before you create a Channel, make sure that you meet the following requirements:
An Amazon MSK Provisioned cluster with Express brokers. Standard brokers and Amazon MSK Serverless are not supported.
At least one Kafka topic.
An Amazon S3 bucket for the dead-letter queue (DLQ). This is required.
An IAM service role that the Channel assumes to deliver data.
Data freshness configured between 5 and 15 minutes.
Topic data in JSON, ByteArray, or String format.
A general-purpose Amazon S3 bucket for delivery.
The destination bucket must be in the same AWS Region as your Amazon MSK cluster. Cross-Region delivery is not supported.
Cross-account delivery is supported for the destination bucket only. Your Amazon MSK cluster and the dead-letter queue bucket must be in the same AWS account as the Channel; only the destination bucket can be in a different AWS account.
