Amazon Kinesis Data Firehose
Developer Guide

Troubleshooting Amazon Kinesis Data Firehose

Sometimes you might not see data delivered to your specified destinations. Use the troubleshooting steps in this topic to solve common issues you might encounter while using Kinesis Data Firehose. For further troubleshooting, see Monitoring with CloudWatch Logs.

Data Not Delivered to Amazon S3

Check the following if data is not delivered to your Amazon Simple Storage Service (Amazon S3) bucket.

Data Not Delivered to Amazon Redshift

Check the following if data is not delivered to your Amazon Redshift cluster.

Data is delivered to your S3 bucket before loading into Amazon Redshift. If the data was not delivered to your S3 bucket, see Data Not Delivered to Amazon S3.

  • Check the Kinesis Data Firehose DeliveryToRedshift.Success metric to make sure that Kinesis Data Firehose has attempted to copy data from your S3 bucket to the Amazon Redshift cluster. For more information, see Monitoring Kinesis Data Firehose Using CloudWatch Metrics.

  • Enable error logging if it is not already enabled, and check error logs for delivery failure. For more information, see Monitoring Kinesis Data Firehose Using CloudWatch Logs.

  • Check the Amazon Redshift STL_CONNECTION_LOG table to see if Kinesis Data Firehose is able to make successful connections. In this table, you should be able to see connections and their status based on a user name. For more information, see STL_CONNECTION_LOG in the Amazon Redshift Database Developer Guide.

  • If the previous check shows that connections are being established, check the Amazon Redshift STL_LOAD_ERRORS table to verify the reason of the COPY failure. For more information, see STL_LOAD_ERRORS in the Amazon Redshift Database Developer Guide.

  • Make sure that the Amazon Redshift configuration in your Kinesis Data Firehose delivery stream is accurate and valid.

  • Make sure that the IAM role that is specified in your Kinesis Data Firehose delivery stream can access the S3 bucket that Amazon Redshift copies data from, and the Lambda function for data transformation (if data transformation is enabled). For more information, see Grant Kinesis Data Firehose Access to an Amazon S3 Destination.

  • If your Amazon Redshift cluster is in a virtual private cloud (VPC), make sure that the cluster allows access from Kinesis Data Firehose IP addresses. For more information, see Grant Kinesis Data Firehose Access to an Amazon Redshift Destination .

  • Make sure that the Amazon Redshift cluster is publicly available.

  • If you're using data transformation, make sure that your Lambda function never returns responses whose payload size exceeds 6 MB. For more information, see Amazon Kinesis Data Firehose Data Transformation.

Data Not Delivered to Amazon Elasticsearch Service

Check the following if data is not delivered to your Elasticsearch domain.

Data can be backed up to your S3 bucket concurrently. If data was not delivered to your S3 bucket, see Data Not Delivered to Amazon S3.

Data Not Delivered to Splunk

Check the following if data is not delivered to your Splunk endpoint.

  • If you use an AWS load balancer, make sure that it is a Classic Load Balancer. Kinesis Data Firehose supports neither Application Load Balancers nor Network Load Balancers. Also, enable duration-based sticky sessions with cookie expiration disabled. For information about how to do this, see Duration-Based Session Stickiness.

  • Review the Splunk platform requirements. The Splunk Add-on for Kinesis Data Firehose requires Splunk platform version 6.6.X or later. For more information, see Splunk Add-on for Amazon Kinesis Firehose.

  • If you have a proxy (Elastic Load Balancing or other) between Kinesis Data Firehose and the HTTP Event Collector (HEC) node, enable sticky sessions to support HEC acknowledgements (ACKs).

  • Make sure that you are using a valid HEC token.

  • Ensure that the HEC token is enabled. See Enable and disable Event Collector tokens.

  • Check to see whether the data that you're sending to Splunk is formatted correctly. See Format events for HTTP Event Collector.

  • Make sure that the HEC token and input event are configured with a valid index.

  • When an upload to Splunk fails due to a server error from the HEC node, the request is automatically retried. If all retries fail, the data gets backed up to Amazon S3. Check to see if your data appears in Amazon S3, which is an indication of such a failure.

  • Make sure that you've enabled indexer acknowledgment on your HEC token. See Enable indexer acknowledgement.

  • Increase the value of HECAcknowledgmentTimeoutInSeconds in the Splunk destination configuration of your Kinesis Data Firehose delivery stream.

  • Increase the value of DurationInSeconds under RetryOptionsin the Splunk destination configuration of your Kinesis Data Firehose delivery stream.

  • Check your HEC health.

  • If you're using data transformation, make sure that your Lambda function never returns responses whose payload size exceeds 6 MB. For more information, see Amazon Kinesis Data Firehose Data Transformation.

  • Make sure that the Splunk parameter named ackIdleCleanup is set to true. It's false by default. To set this parameter to true, do the following:

    • For a managed Splunk Cloud deployment, submit a case using the Splunk Support Portal. In this case, ask Splunk Support to enable the HTTP event collector, set ackIdleCleanup to true in inputs.conf, and create or modify a load balancer to use with this add-on.

    • For a distributed Splunk Enterprise deployment, set the ackIdleCleanup parameter to true in the inputs.conf file. This file is located under $SPLUNK_HOME/etc/apps/splunk_httpinput/local/ for *nix users and under %SPLUNK_HOME%\etc\apps\splunk_httpinput\local\ for Windows users.

    • For a single-instance Splunk Enterprise deployment, set the ackIdleCleanup parameter to true in the inputs.conf file. This file is located under $SPLUNK_HOME/etc/apps/splunk_httpinput/local/ for *nix users and under %SPLUNK_HOME%\etc\apps\splunk_httpinput\local\ for Windows users.

  • See Troubleshoot the Splunk Add-on for Amazon Kinesis Firehose.

Delivery Stream Not Available as a Target for CloudWatch Logs, CloudWatch Events, or AWS IoT Action

Some AWS services can only send messages and events to a Kinesis Data Firehose delivery stream that is in the same Region. Verify that your Kinesis Data Firehose delivery stream is located in the same Region as your other services.