Auto-expanding storage for an Amazon MSK Cluster - Amazon Managed Streaming for Apache Kafka

Auto-expanding storage for an Amazon MSK Cluster

You can configure Amazon Managed Streaming for Apache Kafka to automatically expand your cluster's storage in response to increased usage using Application Auto-Scaling policies. Your auto-scaling policy sets the target disk utilization and the maximum scaling capacity.

Note

The Amazon MSK service does not reduce cluster storage in response to reduced usage. Amazon MSK does not support decreasing the size of storage volumes. If you need to reduce the size of your cluster storage, you must migrate your existing cluster to a cluster with smaller storage. For information about migrating a cluster, see Migration.

MSK Storage auto-expansion details

Your auto scaling policy defines the following parameters for your cluster:

  • Storage Utilization Target: The storage utilization threshold that Amazon MSK uses to trigger an auto-scaling operation. You can set the utilization target between 10% and 80% of the current storage capacity. We recommend that you set the Storage Utilization Target between 50% and 60%.

  • Maximum Storage Capacity: This setting is the maximum scaling limit that Amazon MSK can set your broker storage. You can set the maximum storage capacity up to 16 TiB per broker. For more information, see Amazon MSK Limits.

When the service detects that your Maximum Disk Utilization metric is equal to or greater than the Storage Utilization Target setting, it will increase your storage capacity automatically. Amazon MSK first expands your cluster storage by an amount equal to the larger of two numbers: 10 GiB and 10% of current storage. For example, if you have 1000 GiB, that amount is 100 GiB. Further scaling operations increase storage by a greater amount. The service checks your storage utilization every minute.

Note

Auto scaling has a cool-down period of at least six hours between events. Even though the operation makes additional storage available right away, the service performs optimizations on your cluster that can take up to 24 hours. The duration of these optimizations is proportional to your storage size. To repeatedly expand your storage capacity more quickly, set the storage capacity manually using the UpdateBrokerStorage operation. For information about right-sizing your storage, see Right-size your cluster.

You can check to determine if auto scaling operations occurred using the ListClusterOperations operation.

Setting up auto-expansion for your Amazon MSK cluster

You can use the Amazon MSK console or the Amazon MSK API to implement auto-expanding storage.

Note

You can't implement auto-expansion when you create a cluster. You must first create the cluster, and then create and enable an auto-expansion policy for it. However, you can create the policy while the Amazon MSK service creates your cluster.

Setting up auto-expansion using the AWS Management Console

  1. Sign in to the AWS Management Console, and open the Amazon MSK console at https://console.aws.amazon.com/msk/home?region=us-east-1#/home/.

  2. Navigate to the Cluster Details page.

  3. Create and name an auto-scaling policy. Specify the storage utilization target, the maximum storage capacity, and the target metric (KafkaDataLogsDiskUsed).

  4. Save and enable the new policy.

When you save and enable the new policy, the policy becomes active for the cluster. Amazon MSK then expands the cluster's storage when the storage utilization target is reached.

Setting up auto-expansion using the CLI

  1. Use the RegisterScalableTarget command to register a storage utilization target.

  2. Use the PutScalingPolicy command to create an auto-expansion policy.

Setting up auto-expansion using the API

  1. Use the RegisterScalableTarget API to register a storage utilization target.

  2. Use the PutScalingPolicy API to create an auto-expansion policy.