View a markdown version of this page

Partitioning your output - Amazon Managed Streaming for Apache Kafka

Partitioning your output

  • By Kafka partition — include !{partition-id} as a path segment, for example !{topic-name}/!{partition-id}/....

  • By time — include time variables as path segments, for example !{yyyy}/!{MM}/!{dd}/!{HH}/.... Avoid minute-level (!{mm}) granularity for high-throughput topics — it creates many small objects and can slow delivery and downstream queries.

  • By topic — include !{topic-name} as a path segment.

  • Unique object name (required) — each delivered object must have a unique key, so the final path segment must include a batch token: !{sequence-number} or !{kafka-offset}.