Menu
Amazon Kinesis Streams
Developer Guide

Troubleshooting Amazon Kinesis Streams Producers

The following sections offer solutions to some common problems you may find while working with Amazon Kinesis Streams producers.

Producer Application is Writing at a Slower Rate Than Expected

The most common reasons for write throughput being slower than expected are as follows.

Service Limits Exceeded

To find out if service limits are being exceeded, check to see if your producer is throwing throughput exceptions from the service, and validate what API operations are being throttled. Keep in mind that there are different limits based on the call, see Amazon Kinesis Streams Limits. For example, in addition to the shard-level limits for writes and reads that are most commonly known, there are the following stream-level limits:

The operations CreateStream, DeleteStream, ListStreams, GetShardIterator, and MergeShards are limited to 5 calls per second. The DescribeStream operation is limited to 10 calls per second.

If these calls aren't the issue, make sure you've selected a partition key that allows you to distribute put operations evenly across all shards, and that you don't have a particular partition key that's bumping into the service limits when the rest are not. This requires that you measure peak throughput and take into account the number of shards in your stream. For more information about managing streams, see Managing Kinesis Streams Using Java.

Tip

Remember to round up to the nearest kilobyte for throughput throttling calculations when using the single-record operation PutRecord, while the multi-record operation PutRecords rounds on the cumulative sum of the records in each call. For example, a PutRecords request with 600 records that are 1.1 KB in size will not get throttled.

Producer Optimization

Before you begin optimizing your producer, there are some key tasks to be completed. First, identify your desired peak throughput in terms of record size and records per second. Next, rule out stream capacity as the limiting factor (Service Limits Exceeded). If you've ruled out stream capacity, use the following troubleshooting tips and optimization guidelines for the two common types of producers.

Large Producer

A large producer is usually running from an on-premises server or Amazon EC2 instance. Customers who need higher throughput from a large producer typically care about per-record latency. Strategies for dealing with latency include the following: If the customer can micro-batch/buffer records, use the Kinesis Producer Library (which has advanced aggregation logic), the multi-record operation PutRecords, or aggregate records into a larger file before using the single-record operation PutRecord. If you are unable to batch/buffer, use multiple threads to write to the Kinesis Streams service at the same time. The AWS SDK for Java and other SDKs include async clients that can do this with very little code.

Small Producer

A small producer is usually a mobile app, IoT device, or web client. If it’s a mobile app, we recommend using the PutRecords operation or the Kinesis Recorder in the AWS Mobile SDKs. For more information, see AWS Mobile SDK for Android Getting Started Guide and AWS Mobile SDK for iOS Getting Started Guide. Mobile apps must handle intermittent connections inherently and need some sort of batch put, such as PutRecords. If you are unable to batch for some reason, see the Large Producer information above. If your producer is a browser, the amount of data being generated is typically very small. However, you are putting the put operations on the critical path of the application, which we don’t recommend.