Menu
Amazon Kinesis Firehose
Developer Guide

Monitoring with Amazon CloudWatch Metrics

Kinesis Data Firehose integrates with CloudWatch metrics so that you can collect, view, and analyze CloudWatch metrics for your Kinesis data delivery streams. For example, you can monitor the IncomingBytes and IncomingRecords metrics to keep track of data ingested into Kinesis Data Firehose from data producers.

The metrics that you configure for your Kinesis data delivery streams and agents are automatically collected and pushed to CloudWatch every five minutes. Metrics are archived for two weeks; after that period, the data is discarded.

The metrics collected for Kinesis data delivery streams are free of charge. For information about Kinesis agent metrics, see Monitoring Kinesis Agent Health.

Service-level CloudWatch Metrics

The AWS/Firehose namespace includes the following service-level metrics.

Metric Description
BackupToS3.Bytes

The number of bytes delivered to Amazon S3 for backup over the specified time period. Kinesis Data Firehose emits this metric when data transformation is enabled for Amazon S3 or Amazon Redshift destinations.

Units: Bytes

BackupToS3.DataFreshness

Age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the Amazon S3 bucket for backup. Kinesis Data Firehose emits this metric when data transformation is enabled for Amazon S3 or Amazon Redshift destinations.

Units: Seconds

BackupToS3.Records

The number of records delivered to Amazon S3 for backup over the specified time period. Kinesis Data Firehose emits this metric when data transformation is enabled for Amazon S3 or Amazon Redshift destinations.

Units: Count

BackupToS3.Success

Sum of successful Amazon S3 put commands for backup over sum of all Amazon S3 backup put commands. Kinesis Data Firehose emits this metric when data transformation is enabled for Amazon S3 or Amazon Redshift destinations.

DataReadFromKinesisStream.Bytes

When the data source is a Kinesis data stream, this metric indicates the number of bytes read from that Kinesis data stream. This number includes rereads due to failovers.

Units: Bytes

DataReadFromKinesisStream.Records

When the data source is a Kinesis data stream, this metric indicates the number of records read from that Kinesis data stream. This number includes rereads due to failovers.

Units: Count

DeliveryToElasticsearch.Bytes

The number of bytes indexed to Amazon ES over the specified time period.

Units: Bytes

DeliveryToElasticsearch.Records

The number of records indexed to Amazon ES over the specified time period.

Units: Count

DeliveryToElasticsearch.Success

The sum of the successfully indexed records over the sum of records that were attempted.

DeliveryToRedshift.Bytes

The number of bytes copied to Amazon Redshift over the specified time period.

Units: Bytes

DeliveryToRedshift.Records

The number of records copied to Amazon Redshift over the specified time period.

Units: Count

DeliveryToRedshift.Success

The sum of successful Amazon Redshift COPY commands over the sum of all Amazon Redshift COPY commands.

DeliveryToS3.Bytes

The number of bytes delivered to Amazon S3 over the specified time period.

Units: Bytes

DeliveryToS3.DataFreshness

The age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the S3 bucket.

Units: Seconds

DeliveryToS3.Records

The number of records delivered to Amazon S3 over the specified time period.

Units: Count

DeliveryToS3.Success

The sum of successful Amazon S3 put commands over the sum of all Amazon S3 put commands.

DeliveryToSplunk.Bytes

The number of bytes delivered to Splunk over the specified time period.

Units: Bytes

DeliveryToSplunk.DataFreshness

Age (from getting into Kinesis Firehose to now) of the oldest record in Kinesis Firehose. Any record older than this age has been delivered to Splunk.

Units: Seconds

DeliveryToSplunk.Records

The number of records delivered to Splunk over the specified time period.

Units: Count

DeliveryToSplunk.Success

The sum of the successfully indexed records over the sum of records that were attempted.

IncomingBytes

The number of bytes ingested into the Kinesis Data Firehose stream over the specified time period.

Units: Bytes

IncomingRecords

The number of records ingested into the Kinesis Data Firehose stream over the specified time period.

Units: Count

KinesisMillisBehindLatest

When the data source is a Kinesis data stream, this metric indicates the number of milliseconds that the last read record is behind the newest record in the Kinesis data stream.

Units: Millisecond

API-Level CloudWatch Metrics

The AWS/Firehose namespace includes the following API-level metrics.

Metric Description
DescribeDeliveryStream.Latency

The time taken per DescribeDeliveryStream operation, measured over the specified time period.

Units: Milliseconds

DescribeDeliveryStream.Requests

The total number of DescribeDeliveryStream requests.

Units: Count

ListDeliveryStreams.Latency

The time taken per ListDeliveryStream operation, measured over the specified time period.

Units: Milliseconds

ListDeliveryStreams.Requests

The total number of ListFirehose requests.

Units: Count

PutRecord.Bytes

The number of bytes put to the Kinesis Data Firehose delivery stream using PutRecord over the specified time period.

Units: Bytes

PutRecord.Latency

The time taken per PutRecord operation, measured over the specified time period.

Units: Milliseconds

PutRecord.Requests

The total number of PutRecord requests, which is equal to total number of records from PutRecord operations.

Units: Count

PutRecordBatch.Bytes

The number of bytes put to the Kinesis Data Firehose delivery stream using PutRecordBatch over the specified time period.

Units: Bytes

PutRecordBatch.Latency

The time taken per PutRecordBatch operation, measured over the specified time period.

Units: Milliseconds

PutRecordBatch.Records

The total number of records from PutRecordBatch operations.

Units: Count

PutRecordBatch.Requests

The total number of PutRecordBatch requests.

Units: Count

ThrottledDescribeStream

The total number of times the DescribeStream operation is throttled when the data source is a Kinesis data stream.

Units: Count

ThrottledGetRecords

The total number of times the GetRecords operation is throttled when the data source is a Kinesis data stream.

Units: Count

ThrottledGetShardIterator

The total number of times the GetShardIterator operation is throttled when the data source is a Kinesis data stream.

Units: Count

UpdateDeliveryStream.Latency

The time taken per UpdateDeliveryStream operation, measured over the specified time period.

Units: Milliseconds

UpdateDeliveryStream.Requests

The total number of UpdateDeliveryStream requests.

Units: Count

Data Transformation CloudWatch Metrics

If data transformation with Lambda is enabled, the AWS/Firehose namespace includes the following metrics.

Metric Description
ExecuteProcessing.Duration

The time it takes for each Lambda function invocation performed by Kinesis Data Firehose.

Units: Milliseconds

ExecuteProcessing.Success

The sum of the successful Lambda function invocations over the sum of the total Lambda function invocations.

SucceedProcessing.Records

The number of successfully processed records over the specified time period.

Units: Count

SucceedProcessing.Bytes

The number of successfully processed bytes over the specified time period.

Units: Bytes

Format Conversion CloudWatch Metrics

If format conversion is enabled, the AWS/Firehose namespace includes the following metrics.

Metric Description
SucceedConversion.Records

The number of successfully converted records.

Units: Count

SucceedConversion.Bytes

The size of the successfully converted records.

Units: Bytes

FailedConversion.Records

The number of records that could not be converted.

Units: Count

FailedConversion.Bytes

The size of the records that could not be converted.

Units: Bytes

Dimensions for Kinesis Data Firehose

To filter metrics by delivery stream, use the DeliveryStreamName dimension.

Accessing CloudWatch Metrics for Kinesis Data Firehose

You can monitor metrics for Kinesis Data Firehose using the CloudWatch console, command line, or CloudWatch API. The following procedures show you how to access metrics using these different methods.

To access metrics using the CloudWatch console

  1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.

  2. On the navigation bar, choose a region.

  3. In the navigation pane, choose Metrics.

  4. Choose the Firehose namespace.

  5. Choose Delivery Stream Metrics or Firehose Metrics.

  6. Select a metric to add to the graph.

To access metrics using the AWS CLI

Use the list-metrics and get-metric-statistics commands.

aws cloudwatch list-metrics --namespace "AWS/Firehose"
aws cloudwatch get-metric-statistics --namespace "AWS/Firehose" \ --metric-name DescribeDeliveryStream.Latency --statistics Average --period 3600 \ --start-time 2017-06-01T00:00:00Z --end-time 2017-06-30T00:00:00Z