Quotas and Limits - Amazon Kinesis Data Streams

Quotas and Limits

Amazon Kinesis Data Streams has the following stream and shard quotas and limits.

Quota On-demand mode Provisioned mode

Number of data streams

There is no upper quota on the number of streams within your AWS account. By default, you can create up to 50 data streams with the on-demand capacity mode. If you require an increase of this quota, please raise a support ticket.

There is no upper quota on the number of streams with the provisioned mode within an account.

Number of shards

There is no upper limit. Number of shards depends on the amount of data ingested and the level of throughput you require. Kinesis Data Streams automatically scales the number of shards in response to changes in data volume and traffic.

There is no upper limit. The default shard quota is 500 shards per AWS account for the following AWS Regions: US East (N. Virginia), US West (Oregon), and Europe (Ireland). For all other regions, the default shard quota is 200 shards per AWS account. To request a shards-per-data stream quota increase, see Requesting a Quota Increase.

Data stream throughput

By default, new data streams created with the on-demand capacity mode have 4 MB/s of write and 8 MB/s of read throughput. As the traffic increases, data streams with the on-demand capacity mode scale up to 200 MB/s of write and 400 MB/s read throughput. If you require an increase to 2 GB/s write and 4 GB/s read capacity, submit a support ticket

There is no upper limit. Maximum throughput depends on the number of shards provisioned for the stream. Each shard can support up to 1 MB/sec or 1,000 records/sec write throughput or up to 2 MB/sec or 2,000 records/sec read throughput. If you need more ingest capacity, you can easily scale up the number of shards in the stream using the AWS Management Console or the UpdateShardCount API.

Data payload size

The maximum size of the data payload of a record before base64-encoding is up to 1 MB.

GetRecords transaction size

GetRecords can retrieve up to 10 MB of data per call from a single shard, and up to 10,000 records per call. Each call to GetRecords is counted as one read transaction. Each shard can support up to five read transactions per second. Each read transaction can provide up to 10,000 records with an upper quota of 10 MB per transaction.

Data read rate per shard

Each shard can support up to a maximum total data read rate of 2 MB per second via GetRecords. If a call to GetRecords returns 10 MB, subsequent calls made within the next 5 seconds throw an exception.

Number of registered consumers per data stream

You can create up to 20 registered consumers (Enhanced Fan-out Limit) for each data stream.

Switching between provisioned and on-demand modes

For each data stream in your AWS account, you can switch between the on-demand and provisioned capacity modes twice within 24 hours. 

API Limits

Like most AWS APIs, Kinesis Data Streams API operations are rate-limited. The following limits apply per AWS account per region. For more information on Kinesis Data Streams APIs, see the Amazon Kinesis API Reference.

KDS Control Plane API Limits

The following section describes limits for the KDS control plane APIs. KDS control plane APIs enable you to create and manage your data streams. These limits apply per AWS account per region.

Control Plane API Limits
API API call limit Per Account/Stream  Description
AddTagsToStream 5 transactions per second (TPS) Per Account 50 tags per data stream
CreateStream 5 TPS Per Account

There is no upper quota on the number of streams you can have in an account. You receive a LimitExceededException when making a CreateStream request when you try to do one of the following:

  • Have more than five streams in the CREATING state at any point in time.

  • Create more shards than are authorized for your account.

DecreaseStreamRetentionPeriod 5 TPS Per Stream The minimum value of a data stream's retention period is 24 hours.
DeleteResourcePolicy 5 TPS Per Account If you require an increase of this limit, raise a Support ticket.
DeleteStream 5 TPS Per Account
DeregisterStreamConsumer 5 TPS Per Stream
DescribeLimits 1 TPS Per Account
DescribeStream 10 TPS Per Account
DescribeStreamConsumer 20 TPS Per Stream
DescribeStreamSummary 20 TPS Per Account
DisableEnhancedMonitoring 5 TPS Per Stream
EnableEnhancedMonitoring 5 TPS Per Stream
GetResourcePolicy 5 TPS Per Account If you require an increase of this limit, raise a Support ticket.
IncreaseStreamRetentionPeriod 5 TPS Per Stream The maximum value of a stream's retention period is 8760 hours (365 days).
ListShards 1000 TPS Per Stream
ListStreamConsumers 5 TPS Per Stream
ListStreams 5 TPS Per Account
ListTagsForStream 5 TPS Per Stream
MergeShards 5 TPS Per Stream Only applicable for provisioned.
PutResourcePolicy 5 TPS Per Account If you require an increase of this limit, raise a Support ticket.
RegisterStreamConsumer 5 TPS Per Stream You can register up to 20 consumers per data stream. A given consumer can only be registered with one data stream at a time. Only 5 consumers can be created simultaneously. In other words, you cannot have more than 5 consumers in a CREATING status at the same time. Registering a 6th consumer while there are 5 in a CREATING
RemoveTagsFromStream 5 TPS Per Stream
SplitShard 5 TPS Per Stream Only applicable for provisioned
StartStreamEncryption Per Stream You can successfully apply a new AWS KMS key for server-side encryption 25 times in a rolling 24-hour period.
StopStreamEncryption Per Stream You can successfully disable server-side encryption 25 times in a rolling 24-hour period.
UpdateShardCount Per Stream Only applicable for provisioned. Default limit on number of shards is 10,000. There are additional limits on this API. For more information, see UpdateShardCount.
UpdateStreamMode Per stream

For each data stream in your AWS account, you can switch between the on-demand and provisioned capacity modes twice within 24 hours.

KDS Data Plane API Limits

The following section describes the limits for the KDS data plane APIs. KDS data plane APIs enable you to use your data streams for collecting and processing data records in real time. These limits apply per shard within your data streams.

Data Plane API limits
API API call limit Payload limit Additional details
GetRecords 5 TPS The maximum number of records that can be returned per call is 10,000. The maximum size of data that GetRecords can return is 10 MB. If a call returns this amount of data, subsequent calls made within the next 5 seconds throw ProvisionedThroughputExceededException. If there is insufficient provisioned throughput on the stream, subsequent calls made within the next 1 second throws ProvisionedThroughputExceededException.
GetShardIterator 5 TPS A shard iterator expires 5 minutes after it is returned to the requester. If a GetShardIterator request is made too often, you receive a ProvisionedThroughputExceededException.
PutRecord 1000 TPS Each shard can support writes up to 1,000 records per second, up to a maximum data write total of 1 MB per second.
PutRecords Each PutRecords request can support up to 500 records. Each record in the request can be as large as 1 MB, up to a limit of 5 MB for the entire request, including partition keys. Each shard can support writes up to 1,000 records per second, up to a maximum data write total of 1 MB per second.
SubscribeToShard You can make one call to SubscribeToShard per second per registered consumer per shard. If you call SubscribeToShard again with the same ConsumerARN and ShardId within 5 seconds of a successful call, you'll get a ResourceInUseException.

Increasing Quotas

You can use Service Quotas to request an increase for a quota, if the quota is adjustable. Some requests are automatically resolved, while others are submitted to AWS Support. You can track the status of a quota increase request that is submitted to AWS Support. Requests to increase service quotas do not receive priority support. If you have an urgent request, contact AWS Support. For more information, see What Is Service Quotas?

To request a service quota increase, follow the procedure outlined in Requesting a Quota Increase.