Best Practices

This topic outlines some best practices to follow when using Amazon MSK.

Right-size Your Cluster

To determine the right size for your MSK cluster and understand costs, see the MSK Sizing and Pricing spreadsheet. This spreadsheet provides an estimate for sizing an MSK cluster and the associated costs of Amazon MSK compared to a similar, self-managed, EC2-based Apache Kafka cluster. For more information about the input parameters in the spreadsheet, hover over the parameter descriptions. This spreadsheet was the result of running a test workload with three producers and three consumers, and ensuring that P99 write latencies were below 100 ms. This might not reflect your workload or performance expectations. Therefore, we recommend that you test your workloads after provisioning the cluster.

Monitor Disk Space

To avoid running out of disk space for messages, create a CloudWatch alarm that watches the KafkaDataLogsDiskUsed metric. When the value of this metric reaches or exceeds 85%, perform one or more of the following actions:

• Increase broker storage. For information on how to do this, see Scaling Up Broker Storage.

• Reduce the message retention period or log size. For information on how to do that, see Adjust the Data Retention Parameters.

• Delete unused topics.

For information on how to set up and use alarms, see Using Amazon CloudWatch Alarms. For a full list of Amazon MSK metrics, see Monitoring an Amazon MSK Cluster.

Adjust the Data Retention Parameters

Consuming messages doesn't remove them from the log. To free up disk space regularly, you can explicitly specify a retention time period, which is how long messages stay in the log. You can also specify a retention log size. When either the retention time period or the retention log size are reached, Apache Kafka starts removing inactive segments from the log.

To specify a retention policy at the cluster level, set one or more of the following parameters: log.retention.hours, log.retention.minutes, log.retention.ms, or log.retention.bytes. For more information, see Custom MSK Configurations.

You can also specify retention parameters at the topic level:

• To specify a retention time period per topic, use the following command.

  kafka-configs.sh --zookeeper ZooKeeperConnectionString --alter --entity-type topics --entity-name TopicName --add-config retention.ms=DesiredRetentionTimePeriod

• To specify a retention log size per topic, use the following command.

  kafka-configs.sh --zookeeper ZooKeeperConnectionString --alter --entity-type topics --entity-name TopicName --add-config retention.bytes=DesiredRetentionLogSize

The retention parameters that you specify at the topic level take precedence over cluster-level parameters.

Don't Add Non-MSK Brokers

If you use Apache ZooKeeper commands to add brokers, these brokers don't get added to your MSK cluster, and your Apache ZooKeeper will contain incorrect information about the cluster. This might result in data loss. For supported cluster operations, see Amazon MSK: How It Works.

Enable In-Transit Encryption

For information about encryption in transit and how to enable it, see Encryption in Transit.

Reassign Partitions

To move partitions to different brokers on the same cluster, you can use the partition reassignment tool named kafka-reassign-partitions.sh. For example, after you add new brokers to expand a cluster, you can rebalance that cluster by reassigning partitions to the new brokers. For information about how to add brokers to a cluster, see Expanding an Amazon MSK Cluster. For information about the partition reassignment tool, see Expanding your cluster in the Apache Kafka documentation.