KRaft mode - Amazon Managed Streaming for Apache Kafka

KRaft mode

Amazon MSK introduced support for KRaft (Apache Kafka Raft) in Kafka version 3.7.x. The Apache Kafka community developed KRaft to replace Apache ZooKeeper for metadata management in Apache Kafka clusters. In KRaft mode, cluster metadata is propagated within a group of Kafka controllers, which are part of the Kafka cluster, instead of across ZooKeeper nodes. KRaft controllers are included at no additional cost to you, and require no additional setup or management from you. See KIP-500 for more information about KRaft.

Here are some points to note about KRaft mode on MSK:

  • KRaft mode is only available for new clusters. You cannot switch metadata modes once the cluster is created.

  • On the MSK console, you can create a Kraft-based cluster by choosing Kafka version 3.7.x and selecting the KRaft checkbox in the cluster creation window.

  • To create a cluster in KRaft mode using the MSK API CreateCluster or CreateClusterV2 operations, you should use 3.7.x.kraft as the version. Use 3.7.x as the version to create a cluster in ZooKeeper mode.

  • The number of partitions per broker is the same on KRaft and ZooKeeper based clusters. However, KRaft allows you to host more partitions per cluster by provisioning more brokers in a cluster.

  • There are no API changes required to use KRaft mode on Amazon MSK. However, if your clients still use the --zookeeper connection string today, you should update your clients to use the --bootstrap-server connection string to connect to your cluster. The --zookeeper flag is deprecated in Apache Kafka version 2.5 and is removed starting with Kafka version 3.0. We therefore recommend you use recent Apache Kafka client versions and the --bootstrap-server connection string for all connections to your cluster.

  • ZooKeeper mode continues to be available for all released versions where zookeeper is also supported by Apache Kafka. See Supported Apache Kafka versions for details on the end of support for Apache Kafka versions and future updates.

  • You should check that any tools you use are capable of using Kafka Admin APIs without ZooKeeper connections. Refer to Using LinkedIn's Cruise Control for Apache Kafka with Amazon MSK for updated steps to connect your cluster to Cruise Control. Cruise Control also has instructions for running Cruise Control without ZooKeeper.

  • You do not need to access your cluster's KRaft controllers directly for any administrative actions. However, if you are using open monitoring to collect metrics, you also need the DNS endpoints of your controllers in order to collect some non-controller related metrics about your cluster. You can get these DNS endpoints from the MSK Console or using the ListNodes API operation. See Open monitoring with Prometheus for updated steps for setting up open-monitoring for KRaft-based clusters.

  • There are no additional CloudWatch metrics you need to monitor for KRaft mode clusters over ZooKeeper mode clusters. MSK manages the KRaft controllers used in your clusters.

  • You can continue managing ACLs using in KRaft mode clusters using the --bootstrap-server connection string. You should not use the --zookeeper connection string to manage ACLs. See Apache Kafka ACLs.

  • In KRaft mode, your cluster’s metadata is stored on KRaft controllers within Kafka and not external ZooKeeper nodes. Therefore, you don't need to control access to controller nodes separately as you do with ZooKeeper nodes.