Amazon MSK quota - Amazon Managed Streaming for Apache Kafka

Amazon MSK quota

Your AWS account has default quotas for Amazon MSK. Unless otherwise stated, each per-account quota is region-specific within your AWS account.

Amazon MSK quota

  • Up to 90 brokers per account. 30 brokers per ZooKeeper mode cluster. 60 brokers per KRaft mode cluster. To request higher quota, go to the Service Quotas console.

  • A minimum of 1 GiB of storage per broker.

  • A maximum of 16384 GiB of storage per broker.

  • A cluster that uses IAM access control can have up to 3000 TCP connections per broker at any given time. To increase this limit, you can adjust the listener.name.client_iam.max.connections or the listener.name.client_iam_public.max.connections configuration property using the Kafka AlterConfig API or the kafka-configs.sh tool. It's important to note that increasing either property to a high value can result in unavailability.

  • Limits on TCP connections. With connection rate bursts enabled, MSK allows 100 connections per second. The exception is the kafka.t3.small instance type, which is allowed 4 connections per second with connection rate bursts enabled. Older clusters that don't have connection rate bursts enabled will have the feature automatically enabled when the cluster is patched.

    To handle retries on failed connections, you can set the reconnect.backoff.ms configuration parameter on the client side. For example, if you want a client to retry connections after 1 second, set reconnect.backoff.ms to 1000. For more information, see reconnect.backoff.ms in the Apache Kafka documentation.

  • Up to 100 configurations per account. To request a limit increase through Service Quotas, go to the Service Quotas console.

  • A maximum of 50 revisions per configuration.

  • To update the configuration or the Apache Kafka version of an MSK cluster, first ensure the number of partitions per broker is under the limits described in Right-size your cluster: Number of partitions per broker.

MSK Replicator quotas

  • A maximum of 15 MSK Replicators per account.

  • MSK Replicator only replicates up to 750 topics in sorted order. If you need to replicate more topics, we recommend that you create a separate Replicator. Go to the Service Quotas console, if you need support for more than 750 topics per Replicator. You can monitor the number of topics being replicated using the "TopicCount" metric.

  • A maximum ingress throughput of 1GB per second per MSK Replicator. Request a higher quota by going through the Service Quotas console.

  • MSK Replicator Record Size - A maximum of 10MB record size (message.max.bytes). Request a higher quota by going through the Service Quotas console.

MSK Serverless quota

Note

If you experience any issues with quota limits, go to the Service Quotas console.

Limits are per cluster, unless otherwise stated.
Dimension Quota Quota violation result

Maximum ingress throughput

200 MBps

Slowdown with throttle duration in response

Maximum egress throughput 400 MBps

Slowdown with throttle duration in response

Maximum retention duration Unlimited N/A

Maximum number of client connections

3000 Connection close
Maximum connection attempts 100 per second Connection close

Maximum message size

8 MB Request fails with ErrorCode: INVALID_REQUEST
Maximum request rate 15,000 per second Slowdown with throttle duration in response

Maximum rate of topic management APIs requests rate

2 per second Slowdown with throttle duration in response

Maximum fetch bytes per request

55 MB

Request fails with ErrorCode: INVALID_REQUEST

Maximum number of consumer groups

500 JoinGroup request fails
Maximum number of partitions (leaders) 2400 for non-compacted topics. 120 for compacted topics. To request a quota adjustment, go to the AWS console Support Center and create a support case. Request fails with ErrorCode: INVALID_REQUEST
Maximum rate of partition creation and deletion 250 in 5 minutes Request fails with ErrorCode: THROUGHPUT_QUOTA_EXCEEDED

Maximum ingress throughput per partition

5 MBps Slowdown with throttle duration in response

Maximum egress throughput per partition

10 MBps Slowdown with throttle duration in response

Maximum partition size (for compacted topics)

250 GB Request fails with ErrorCode: THROUGHPUT_QUOTA_EXCEEDED
Maximum number of client VPCs per serverless cluster 5
Maximum number of serverless clusters per account 10. To request a quota adjustment, go to the Service Quotas console.

MSK Connect quota

  • Up to 100 custom plugins.

  • Up to 100 worker configurations.

  • Up to 60 connect workers. If a connector is set up to have auto scaled capacity, then the maximum number of workers that the connector is set up to have is the number MSK Connect uses to calculate the quota for the account.

  • Up to 10 workers per connector.

To request higher quota for MSK Connect, go to the Service Quotas console.