When Firehose delivers data to Amazon S3, S3 object key name follows the format <evaluated prefix><suffix>, where the suffix has the format <Firehose stream name>-<Firehose stream version>-<year>-<month>-<day>-<hour>-<minute>-<second>-<uuid><file extension> <Firehose stream version> begins with 1 and increases by 1 for every configuration change of Firehose stream. You can change Firehose stream configurations (for example, the name of the S3 bucket, buffering hints, compression, and encryption). You can do so by using the Firehose console or the UpdateDestination API operation.
For <evaluated prefix>, Firehose adds a default time prefix in the format YYYY/MM/dd/HH
.
This prefix creates a logical hierarchy in the bucket, where each forward slash (/) creates a level in the hierarchy.
You can modify this structure by specifying a custom prefix that includes expressions that are evaluated at runtime.
For information about how to specify a custom prefix, see Custom Prefixes for Amazon Simple Storage Service Objects.
By default, the time zone used for time prefix and suffix is in UTC, but you can change it to a time zone that you prefer. For example, to use Japan Standard Time instead of UTC, you can configure the time zone to Asia/Tokyo in the AWS Management Console or in API parameter setting (CustomTimeZone). The following list contains time zones that Firehose supports for S3 prefix configuration.
Following is a list of time zones that Firehose supports for S3 prefix configuration.
You cannot change the suffix field except <file extension>. When you enable data format conversion or compression, Firehose will append a file extension based on the configuration. The following table explains the default file extension appended by Firehose:
Configuration | File extension |
---|---|
Data Format Conversion: Parquet | .parquet |
Data Format Conversion: ORC | .orc |
Compression: Gzip | .gz |
Compression: Zip | .zip |
Compression: Snappy | .snappy |
Compression: Hadoop-Snappy | .hsnappy |
You can also specify a file extension that you prefer in the Firehose console or API. File extension must start with a period (.) and can contain allowed characters: 0-9a-z!-_.*‘(). File extension cannot exceed 128 characters.
Note
When you specify a file extension, it will override the default file extension that Firehose adds when data format conversion or compression is enabled.