Sign In to the Console
Amazon Kinesis Data Firehose
Developer Guide

AWS Documentation » Amazon Kinesis Firehose » Developer Guide » Creating an Amazon Kinesis Data Firehose Delivery Stream » Choose destination

Choose destination

This topic describes the Choose destination page of the Create Delivery Stream wizard in Amazon Kinesis Data Firehose.

Kinesis Data Firehose can send records to Amazon Simple Storage Service (Amazon S3), Amazon Redshift, or Amazon Elasticsearch Service (Amazon ES).

Choose Amazon S3 for Your Destination

This section describes options for using Amazon S3 for your destination.

To choose Amazon S3 for your destination

  • On the Choose destination page, enter values for the following fields:

    Destination

    Choose Amazon S3.

    Destination S3 bucket

    Choose an S3 bucket that you own where the streaming data should be delivered. You can create a new S3 bucket or choose an existing one.

    Destination S3 bucket prefix

    (Optional) To use the default prefix for Amazon S3 objects, leave this option blank. Kinesis Data Firehose automatically uses a prefix in "YYYY/MM/DD/HH" UTC time format for delivered Amazon S3 objects. You can add to the start of this prefix. For more information, see Amazon S3 Object Name Format.

    Source record S3 backup

    Choose Disabled to disable source record backup. If you enable data transformation with AWS Lambda, you can enable source record backup to deliver untransformed incoming data to a separate S3 bucket. You can add to the start of the "YYYY/MM/DD/HH" UTC time prefix that is generated by Kinesis Data Firehose. You cannot disable source record backup after you enable it.

Choose Amazon Redshift for Your Destination

This section describes options for using Amazon Redshift for your destination.

To choose Amazon Redshift for your destination

  • On the Choose destination page, enter values for the following fields:

    Destination

    Choose Amazon Redshift.

    Cluster

    The Amazon Redshift cluster to which S3 bucket data is copied. Configure the Amazon Redshift cluster to be publicly accessible and unblock Kinesis Data Firehose IP addresses. For more information, see Grant Kinesis Data Firehose Access to an Amazon Redshift Destination .

    User name

    An Amazon Redshift user with permissions to access the Amazon Redshift cluster. This user must have the Amazon Redshift INSERT permission for copying data from the S3 bucket to the Amazon Redshift cluster.

    Password

    The password for the user who has permissions to access the cluster.

    Database

    The Amazon Redshift database to where the data is copied.

    Table

    The Amazon Redshift table to where the data is copied.

    Columns

    (Optional) The specific columns of the table to which the data is copied. Use this option if the number of columns defined in your Amazon S3 objects is less than the number of columns within the Amazon Redshift table.

    Intermediate S3 bucket

    Kinesis Data Firehose delivers your data to your S3 bucket first and then issues an Amazon Redshift COPY command to load the data into your Amazon Redshift cluster. Specify an S3 bucket that you own where the streaming data should be delivered. Create a new S3 bucket, or choose an existing bucket that you own.

    Kinesis Data Firehose doesn't delete the data from your S3 bucket after loading it to your Amazon Redshift cluster. You can manage the data in your S3 bucket using a lifecycle configuration. For more information, see Object Lifecycle Management in the Amazon Simple Storage Service Developer Guide.

    Intermediate S3 bucket prefix

    (Optional) To use the default prefix for Amazon S3 objects, leave this option blank. Kinesis Data Firehose automatically uses a prefix in "YYYY/MM/DD/HH" UTC time format for delivered Amazon S3 objects. You can add to the start of this prefix. For more information, see Amazon S3 Object Name Format.

    COPY options

    Parameters that you can specify in the Amazon Redshift COPY command. These might be required for your configuration. For example, "GZIP" is required if Amazon S3 data compression is enabled. "REGION" is required if your S3 bucket isn't in the same AWS Region as your Amazon Redshift cluster. For more information, see COPY in the Amazon Redshift Database Developer Guide.

    COPY command

    The Amazon Redshift COPY command. For more information, see COPY in the Amazon Redshift Database Developer Guide.

    Retry duration

    Time duration (0–7200 seconds) for Kinesis Data Firehose to retry if data COPY to your Amazon Redshift cluster fails. Kinesis Data Firehose retries every 5 minutes until the retry duration ends. If you set the retry duration to 0 (zero) seconds, Kinesis Data Firehose does not retry upon a COPY command failure.

    Source record S3 backup

    If you enable data transformation using AWS Lambda, you can enable source record backup to deliver untransformed incoming data to a separate S3 bucket. You cannot disable source record backup after you enable it.

    Backup S3 bucket

    The S3 bucket to receive the untransformed data.

    Backup S3 bucket prefix

    To use the default prefix for source record backup, leave this option blank. Kinesis Data Firehose automatically uses a prefix in "YYYY/MM/DD/HH" UTC time format for delivered Amazon S3 objects. You can add to the start of this prefix. For more information, see Amazon S3 Object Name Format. This value is optional.

Choose Amazon ES for Your Destination

This section describes options for using Amazon ES for your destination.

To choose Amazon ES for your destination

  1. On the Choose destination page, enter values for the following fields:

    Destination

    Choose Amazon Elasticsearch Service.

    Domain

    The Amazon ES domain to which your data is delivered.

    Index

    The Elasticsearch index name to be used when indexing data to your Amazon ES cluster.

    Index rotation

    Choose whether and how often the Elasticsearch index should be rotated. If index rotation is enabled, Kinesis Data Firehose appends the corresponding timestamp to the specified index name and rotates. For more information, see Index Rotation for the Amazon ES Destination.

    Type

    The Amazon ES type name to be used when indexing data to your Amazon ES cluster. For Elasticsearch 6.x, there can be only one type per index. If you try to specify a new type for an existing index that already has another type, Kinesis Data Firehose returns an error during runtime.

    Retry duration

    Time duration (0–7200 seconds) for Kinesis Data Firehose to retry if an index request to your Amazon ES cluster fails. Kinesis Data Firehose retries every 5 minutes until the retry duration ends. If you set the retry duration to 0 (zero) seconds, Kinesis Data Firehose does not retry upon an index request failure.

    Backup mode

    You can choose to either back up failed records only or all records. If you choose failed records only, any data that Kinesis Data Firehose can't deliver to your Amazon ES cluster or that your Lambda function can't transform is backed up to the specified S3 bucket. If you choose all records, Kinesis Data Firehose backs up all incoming source data to your S3 bucket concurrently with data delivery to Amazon ES. For more information, see Data Delivery Failure Handling and Data Transformation Failure Handling.

    Backup S3 bucket

    An S3 bucket you own that is the target of the backup data. Create a new S3 bucket, or choose an existing bucket that you own.

    Backup S3 bucket prefix

    (Optional) To use the default prefix for Amazon S3 objects, leave this option blank. Kinesis Data Firehose automatically uses a prefix in "YYYY/MM/DD/HH" UTC time format for delivered Amazon S3 objects. You can add to the start of this prefix. For more information, see Amazon S3 Object Name Format. This value is optional.

  2. Choose Next to go to the Configure settings page.

Choose Splunk for Your Destination

This section describes options for using Splunk for your destination.

To choose Splunk for your destination

  • On the Choose destination page, provide values for the following fields:

    Destination

    Choose Splunk.

    Splunk cluster endpoint

    To determine the endpoint, see Configure Amazon Kinesis Firehose to Send Data to the Splunk Platform in the Splunk documentation.

    Splunk endpoint type

    Choose Raw in most cases. Choose Event if you preprocessed your data using AWS Lambda to send data to different indexes by event type. For information about what endpoint to use, see Configure Amazon Kinesis Firehose to send data to the Splunk platform in the Splunk documentation.

    Authentication token

    To set up a Splunk endpoint that can receive data from Kinesis Data Firehose, see Installation and configuration overview for the Splunk Add-on for Amazon Kinesis Firehose in the Splunk documentation. Save the token that you get from Splunk when you set up the endpoint for this delivery stream, and add it here.

    HEC acknowledgement timeout

    Specify how long Kinesis Data Firehose waits for the index acknowledgement from Splunk. If Splunk doesn’t send the acknowledgment before the timeout is reached, Kinesis Data Firehose considers it a data delivery failure. Kinesis Data Firehose then either retries or backs up the data to your Amazon S3 bucket, depending on the retry duration value that you set.

    Retry duration

    Specify how long Kinesis Data Firehose retries sending data to Splunk.

    After sending data, Kinesis Data Firehose first waits for an acknowledgment from Splunk. If an error occurs or the acknowledgment doesn’t arrive within the acknowledgment timeout period, Kinesis Data Firehose starts the retry duration counter. It keeps retrying until the retry duration expires. After that, Kinesis Data Firehose considers it a data delivery failure and backs up the data to your Amazon S3 bucket.

    Every time that Kinesis Data Firehose sends data to Splunk (either the initial attempt or a retry), it restarts the acknowledgement timeout counter and waits for an acknowledgement from Splunk.

    Even if the retry duration expires, Kinesis Data Firehose still waits for the acknowledgment until it receives it or the acknowledgement timeout period is reached. If the acknowledgment times out, Kinesis Data Firehose determines whether there's time left in the retry counter. If there is time left, it retries again and repeats the logic until it receives an acknowledgment or determines that the retry time has expired.

    If you don't want Kinesis Data Firehose to retry sending data, set this value to 0.

    S3 backup mode

    Choose whether to back up all the events that Kinesis Data Firehose sends to Splunk or only the ones for which delivery to Splunk fails. If you require high data durability, turn on this backup mode for all events. Also consider backing up all events initially, until you verify that your data is getting indexed correctly in Splunk.

    S3 backup bucket

    Choose an existing backup bucket or create a new one.

    S3 backup bucket prefix

    You can specify a prefix for your Amazon S3 backup bucket.