Amazon MSK version support
This topic describes the Amazon MSK version support policy and the procedure for Updating the Apache Kafka version. If you're upgrading your Kafka version, follow the best practices outlined in Best practices for version upgrades.
Amazon MSK version support policy
This section describes the support policy for Amazon MSK supported Kafka versions.
All Kafka versions are supported until they reach their end of support date. For details on end of support dates, see Supported Apache Kafka versions. Upgrade your MSK cluster to the recommended Kafka version or higher version before the end of support date. For details on updating your Apache Kafka version, see Updating the Apache Kafka version. A cluster using a Kafka version after its end of support date is auto-upgraded to the recommended Kafka version. Automatic updates can happen at any time after the end of support date. You will not receive any notification before the update.
MSK will phase out support for newly created clusters that use Kafka versions with published end of support dates.
Updating the Apache Kafka version
You can update an existing MSK cluster to a newer version of Apache Kafka. You can't update it to an older version. When you update the Apache Kafka version of an MSK cluster, also check your client-side software to make sure its version enables you to use the features of the cluster's new Apache Kafka version. Amazon MSK only updates the server software. It doesn't update your clients.
For information about how to make a cluster highly available during an update, see Build highly available clusters.
Important
You can't update the Apache Kafka version for an MSK cluster that exceeds the limits described in Right-size your cluster: Number of partitions per broker.
Updating the Apache Kafka version using the AWS Management Console
Open the Amazon MSK console at https://console.aws.amazon.com/msk/
. -
Choose the MSK cluster on which you want to update the Apache Kafka version.
-
On the Properties tab choose Upgrade in the Apache Kafka version section.
Updating the Apache Kafka version using the AWS CLI
Run the following command, replacing
ClusterArn
with the Amazon Resource Name (ARN) that you obtained when you created your cluster. If you don't have the ARN for your cluster, you can find it by listing all clusters. For more information, see Listing Amazon MSK clusters.aws kafka get-compatible-kafka-versions --cluster-arn
ClusterArn
The output of this command includes a list of the Apache Kafka versions to which you can update the cluster. It looks like the following example.
{ "CompatibleKafkaVersions": [ { "SourceVersion": "2.2.1", "TargetVersions": [ "2.3.1", "2.4.1", "2.4.1.1", "2.5.1" ] } ] }
-
Run the following command, replacing
ClusterArn
with the Amazon Resource Name (ARN) that you obtained when you created your cluster. If you don't have the ARN for your cluster, you can find it by listing all clusters. For more information, see Listing Amazon MSK clusters.Replace
Current-Cluster-Version
with the current version of the cluster. ForTargetVersion
you can specify any of the target versions from the output of the previous command.Important
Cluster versions aren't simple integers. To find the current version of the cluster, use the DescribeCluster operation or the describe-cluster
AWS CLI command. An example version is KTVPDKIKX0DER
.aws kafka update-cluster-kafka-version --cluster-arn
ClusterArn
--current-versionCurrent-Cluster-Version
--target-kafka-versionTargetVersion
The output of the previous command looks like the following JSON.
{ "ClusterArn": "arn:aws:kafka:us-east-1:012345678012:cluster/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2", "ClusterOperationArn": "arn:aws:kafka:us-east-1:012345678012:cluster-operation/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2/0123abcd-abcd-4f7f-1234-9876543210ef" }
-
To get the result of the
update-cluster-kafka-version
operation, run the following command, replacingClusterOperationArn
with the ARN that you obtained in the output of theupdate-cluster-kafka-version
command.aws kafka describe-cluster-operation --cluster-operation-arn
ClusterOperationArn
The output of this
describe-cluster-operation
command looks like the following JSON example.{ "ClusterOperationInfo": { "ClientRequestId": "62cd41d2-1206-4ebf-85a8-dbb2ba0fe259", "ClusterArn": "arn:aws:kafka:us-east-1:012345678012:cluster/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2", "CreationTime": "2021-03-11T20:34:59.648000+00:00", "OperationArn": "arn:aws:kafka:us-east-1:012345678012:cluster-operation/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2/0123abcd-abcd-4f7f-1234-9876543210ef", "OperationState": "UPDATE_IN_PROGRESS", "OperationSteps": [ { "StepInfo": { "StepStatus": "IN_PROGRESS" }, "StepName": "INITIALIZE_UPDATE" }, { "StepInfo": { "StepStatus": "PENDING" }, "StepName": "UPDATE_APACHE_KAFKA_BINARIES" }, { "StepInfo": { "StepStatus": "PENDING" }, "StepName": "FINALIZE_UPDATE" } ], "OperationType": "UPDATE_CLUSTER_KAFKA_VERSION", "SourceClusterInfo": { "KafkaVersion": "2.4.1" }, "TargetClusterInfo": { "KafkaVersion": "2.6.1" } } }
If
OperationState
has the valueUPDATE_IN_PROGRESS
, wait a while, then run thedescribe-cluster-operation
command again. When the operation is complete, the value ofOperationState
becomesUPDATE_COMPLETE
. Because the time required for Amazon MSK to complete the operation varies, you might need to check repeatedly until the operation is complete.
Updating the Apache Kafka version using the API
Invoke the GetCompatibleKafkaVersions operation to get a list of the Apache Kafka versions to which you can update the cluster.
Invoke the UpdateClusterKafkaVersion operation to update the cluster to one of the compatible Apache Kafka versions.
Best practices for version upgrades
To ensure client continuity during the rolling update that is performed as part of the Kafka version upgrade process, review the configuration of your clients and your Apache Kafka topics as follows:
Set the topic replication factor (RF) to a minimum value of
2
for two-AZ clusters and a minimum value of3
for three-AZ clusters. An RF value of2
can lead to offline partitions during patching.Set minimum in-sync replicas (minISR) to a maximum value of
RF - 1
to ensure the partition replica set can tolerate one replica being offline or under-replicated.Configure clients to use multiple broker connection strings. Having multiple brokers in a client’s connection string allows for failover if a specific broker supporting client I/O begins to be patched. For information about how to get a connection string with multiple brokers, see Getting the bootstrap brokers for an Amazon MSK cluster.
We recommend that you upgrade connecting clients to the recommended version or above to benefit from the features available in the new version. Client upgrades are not subject to the end of life (EOL) dates of your MSK cluster's Kafka version, and do not need to be completed by the EOL date. Apache Kafka provides a bi-directional client compatibility policy
that allows older clients to work with newer clusters and vice versa. Kafka clients using versions 3.x.x are likely to come with the following defaults:
acks=all
andenable.idempotence=true
.acks=all
is different from the previous default ofacks=1
and provides extra durability by ensuring that all in-sync replicas acknowledge the produce request. Similarly, the default forenable.idempotence
was previouslyfalse
. The change toenable.idempotence=true
as the default lowers the likelihood of duplicate messages. These changes are considered best practice settings and may introduce a small amount of additional latency that's within normal performance parameters.Use the recommended Kafka version when creating new MSK clusters. Using the recommended Kafka version allows you to benefit from the latest Kafka and MSK features.