Amazon MSK quota - Amazon Managed Streaming for Apache Kafka

Amazon MSK quota

Amazon MSK quota

  • Up to 90 brokers per account and 30 brokers per cluster. To request higher quota, go to the AWS console Support Center and create a support case.

  • A minimum of 1 GiB of storage per broker.

  • A maximum of 16384 GiB of storage per broker.

  • A cluster that uses IAM access control can have up to 3000 TCP connections per broker at any given time. To increase this limit, you can adjust the listener.name.client_iam.max.connections or the listener.name.client_iam_public.max.connections configuration property using the Kafka AlterConfig API or the kafka-configs.sh tool. It's important to note that increasing either property to a high value can result in unavailability.

  • Limits on TCP connections. With connection rate bursts enabled, MSK allows 100 connections per second. The exception is the kafka.t3.small instance type, which is allowed 4 connections per second with connection rate bursts enabled. Older clusters that don't have connection rate bursts enabled will have the feature automatically enabled when the cluster is patched.

    To handle retries on failed connections, you can set the reconnect.backoff.ms configuration parameter on the client side. For example, if you want a client to retry connections after 1 second, set reconnect.backoff.ms to 1000. For more information, see reconnect.backoff.ms in the Apache Kafka documentation.

  • Up to 100 configurations per account. To request a quota adjustment, go to the AWS console Support Center and create a support case.

  • A maximum of 50 revisions per configuration.

  • To update the configuration or the Apache Kafka version of an MSK cluster, first ensure the number of partitions per broker is under the limits described in Right-size your cluster: Number of partitions per broker.

MSK Replicator quotas

  • A maximum of 15 MSK Replicators per account.

  • MSK Replicator only replicates up to 750 topics in sorted order. If you need to replicate more topics, we recommend that you create a separate Replicator. Go to the AWS console Support Center and create a support case if you need support for more than 750 topics per Replicator. You can monitor the number of topics being replicated using the "TopicCount" metric.

  • A maximum ingress throughput of 1GB per second per MSK Replicator. To request higher quota, go to the AWS console Support Center and create a support case.

  • MSK Replicator Record Size - A maximum of 1MB record size (message.max.bytes). To request higher quota, go to the AWS console Support Center and create a support case.

MSK Serverless quota

Note

If you experience any issues with quota limits, contact AWS Support by creating a support case.

Limits are per cluster, unless otherwise stated.
Dimension Quota Quota violation result

Maximum ingress throughput

200 MBps

Slowdown with throttle duration in response

Maximum egress throughput 400 MBps

Slowdown with throttle duration in response

Maximum retention duration Unlimited N/A

Maximum number of client connections

3000 Connection close
Maximum connection attempts 100 per second Connection close

Maximum message size

8 MB Request fails with ErrorCode: INVALID_REQUEST
Maximum request rate 15,000 per second Slowdown with throttle duration in response

Maximum rate of topic management APIs requests rate

2 per second Slowdown with throttle duration in response

Maximum fetch bytes per request

55 MB

Request fails with ErrorCode: INVALID_REQUEST

Maximum number of consumer groups

500 JoinGroup request fails
Maximum number of partitions (leaders) 2400 for non-compacted topics. 120 for compacted topics. To request a quota adjustment, go to the AWS console Support Center and create a support case. Request fails with ErrorCode: INVALID_REQUEST
Maximum rate of partition creation and deletion 250 in 5 minutes Request fails with ErrorCode: THROUGHPUT_QUOTA_EXCEEDED

Maximum ingress throughput per partition

5 MBps Slowdown with throttle duration in response

Maximum egress throughput per partition

10 MBps Slowdown with throttle duration in response

Maximum partition size (for compacted topics)

250 GB Request fails with ErrorCode: THROUGHPUT_QUOTA_EXCEEDED
Maximum number of client VPCs per serverless cluster 5
Maximum number of serverless clusters per account 10. To request a quota adjustment, go to the AWS console Support Center and create a support case.
Maximum number of MSK Replicators per account 15 replicators
Maximum number of topics supported by metrics per MSK Replicator MSK Replicator only replicates up to 750 topics in sorted order. If you need to replicate more topics, we recommend that you create a separate Replicator. Go to the AWS console Support Center and create a support case if you need support for more than 750 topics per Replicator. You can monitor the number of topics being replicated using the "TopicCount" metric.
Maximum ingress throughput per MSK Replicator 1 GB per second per MSK Replicator. To request a quota adjustment, go to the AWS console Support Center and create a support case.
MSK Replicator record size A maximum of 1MB record size (message.max.bytes). To request higher quota, go to the AWS console Support Center and create a support case.

MSK Connect quota

  • Up to 100 custom plugins.

  • Up to 100 worker configurations.

  • Up to 60 connect workers. If a connector is set up to have auto scaled capacity, then the maximum number of workers that the connector is set up to have is the number MSK Connect uses to calculate the quota for the account.

  • Up to 10 workers per connector.

To request higher quota for MSK Connect, go to the AWS console Support Center and create a support case.