Supported Apache Kafka versions - Amazon Managed Streaming for Apache Kafka

Supported Apache Kafka versions

Amazon Managed Streaming for Apache Kafka (Amazon MSK) supports the following Apache Kafka and Amazon MSK versions.

Apache Kafka version 2.8.0

For information about Apache Kafka version 2.8.0, see its release notes on the Apache Kafka downloads site.

Apache Kafka version 2.7.1

For information about Apache Kafka version 2.7.1, see its release notes on the Apache Kafka downloads site.

Apache Kafka version 2.6.2 [Recommended]

For information about Apache Kafka version 2.6.2, see its release notes on the Apache Kafka downloads site.

Apache Kafka version 2.7.0

For information about Apache Kafka version 2.7.0, see its release notes on the Apache Kafka downloads site.

Apache Kafka version 2.6.1

For information about Apache Kafka version 2.6.1, see its release notes on the Apache Kafka downloads site.

Apache Kafka version 2.6.0

For information about Apache Kafka version 2.6.0, see its release notes on the Apache Kafka downloads site.

Apache Kafka version 2.5.1

Apache Kafka version 2.5.1 includes several bug fixes and new features, including encryption in-transit for Apache ZooKeeper and administration clients. Amazon MSK provides TLS ZooKeeper endpoints, which you can query with the DescribeCluster operation.

The output of the DescribeCluster operation includes the ZookeeperConnectStringTls node, which lists the TLS zookeeper endpoints.

The following example shows the ZookeeperConnectStringTls node of the response for the DescribeCluster operation:

"ZookeeperConnectStringTls": "z-3.awskafkatutorialc.abcd123.c3.kafka.us-east-1.amazonaws.com:2182,z-2.awskafkatutorialc.abcd123.c3.kafka.us-east-1.amazonaws.com:2182,z-1.awskafkatutorialc.abcd123.c3.kafka.us-east-1.amazonaws.com:2182"

For information about using TLS encryption with zookeeper, see Using TLS security with Apache ZooKeeper.

For more information about Apache Kafka version 2.5.1, see its release notes on the Apache Kafka downloads site.

Amazon MSK bug-fix version 2.4.1.1

This release is an Amazon MSK-only bug-fix version of Apache Kafka version 2.4.1. This bug-fix release contains a fix for KAFKA-9752, a rare issue that causes consumer groups to continuously rebalance and remain in the PreparingRebalance state. This issue affects clusters running Apache Kafka versions 2.3.1 and 2.4.1. This release contains a community-produced fix that is available in Apache Kafka version 2.5.0.

Note

Amazon MSK clusters running version 2.4.1.1 are compatible with any Apache Kafka client that is compatible with Apache Kafka version 2.4.1.

We recommend that you use MSK bug-fix version 2.4.1.1 for new Amazon MSK clusters if you prefer to use Apache Kafka 2.4.1. You can update existing clusters running Apache Kafka version 2.4.1 to this version to incorporate this fix. For information about upgrading an existing cluster, see Updating the Apache Kafka version.

To work around this issue without upgrading the cluster to version 2.4.1.1, see the Consumer group stuck in PreparingRebalance state section of the guide.

Apache Kafka version 2.4.1 (use 2.4.1.1 instead)

Note

You can no longer create an MSK cluster with Apache Kafka version 2.4.1. Instead, you can use Amazon MSK bug-fix version 2.4.1.1 with clients compatible with Apache Kafka version 2.4.1. And if you already have an MSK cluster with Apache Kafka version 2.4.1, we recommend you update it to use Apache Kafka version 2.4.1.1 instead.

KIP-392 is one of the key Kafka Improvement Proposals that are included in the 2.4.1 release of Apache Kafka. This improvement allows consumers to fetch from the closest replica. To use this feature, set client.rack in the consumer properties to the ID of the consumer's Availability Zone. An example AZ ID is use1-az1. Amazon MSK sets broker.rack to the IDs of the Availability Zones of the brokers. You must also set the replica.selector.class configuration property to org.apache.kafka.common.replica.RackAwareReplicaSelector, which is an implementation of rack awareness provided by Apache Kafka.

When you use this version of Apache Kafka, the metrics in the PER_TOPIC_PER_BROKER monitoring level appear only after their values become nonzero for the first time. For more information about this, see PER_TOPIC_PER_BROKER Level Monitoring.

For information about how to find Availability Zone IDs, see AZ IDs for Your Resource in the AWS Resource Access Manager user guide.

For information about setting configuration properties, see Amazon MSK Configuration.

For more information about KIP-392, see Allow Consumers to Fetch from Closest Replica in the Confluence pages.

For more information about Apache Kafka version 2.4.1, see its release notes on the Apache Kafka downloads site.

Apache Kafka version 2.3.1

For information about Apache Kafka version 2.3.1, see its release notes on the Apache Kafka downloads site.

Apache Kafka version 2.2.1

For information about Apache Kafka version 2.2.1, see its release notes on the Apache Kafka downloads site.

Apache Kafka version 1.1.1 (for existing clusters only)

You can no longer create a new MSK cluster with Apache Kafka version 1.1.1. You can continue to use existing clusters that are configured with Apache Kafka version 1.1.1. For information about Apache Kafka version 1.1.1, see its release notes on the Apache Kafka downloads site.