Updating the Apache Kafka version
You can update an existing MSK cluster to a newer version of Apache Kafka. You can't update it to an older version. When you update the Apache Kafka version of an MSK cluster, also check your client-side software to make sure its version enables you to use the features of the cluster's new Apache Kafka version. Amazon MSK only updates the server software. It doesn't update your clients.
For information about how to make a cluster highly available during an update, see Build highly available clusters.
You can't update the Apache Kafka version for an MSK cluster that exceeds the limits described in Right-size your cluster: Number of partitions per broker.
Updating the Apache Kafka version using the AWS Management Console
Open the Amazon MSK console at https://console.aws.amazon.com/msk/
. -
Choose the MSK cluster on which you want to update the Apache Kafka version.
-
On the Details tab choose Upgrade the Apache Kafka version.
Updating the Apache Kafka version using the AWS CLI
Run the following command, replacing
ClusterArn
with the Amazon Resource Name (ARN) that you obtained when you created your cluster. If you don't have the ARN for your cluster, you can find it by listing all clusters. For more information, see Listing Amazon MSK clusters.aws kafka get-compatible-kafka-versions --cluster-arn
ClusterArn
The output of this command includes a list of the Apache Kafka versions to which you can update the cluster. It looks like the following example.
{ "CompatibleKafkaVersions": [ { "SourceVersion": "2.2.1", "TargetVersions": [ "2.3.1", "2.4.1", "2.4.1.1", "2.5.1" ] } ] }
-
Run the following command, replacing
ClusterArn
with the Amazon Resource Name (ARN) that you obtained when you created your cluster. If you don't have the ARN for your cluster, you can find it by listing all clusters. For more information, see Listing Amazon MSK clusters.Replace
Current-Cluster-Version
with the current version of the cluster. ForTargetVersion
you can specify any of the target versions from the output of the previous command.Important Cluster versions aren't simple integers. To find the current version of the cluster, use the DescribeCluster operation or the describe-cluster
AWS CLI command. An example version is KTVPDKIKX0DER
.aws kafka update-cluster-kafka-version --cluster-arn
ClusterArn
--current-versionCurrent-Cluster-Version
--target-kafka-versionTargetVersion
The output of the previous command looks like the following JSON.
{ "ClusterArn": "arn:aws:kafka:us-east-1:012345678012:cluster/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2", "ClusterOperationArn": "arn:aws:kafka:us-east-1:012345678012:cluster-operation/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2/0123abcd-abcd-4f7f-1234-9876543210ef" }
-
To get the result of the
update-cluster-kafka-version
operation, run the following command, replacingClusterOperationArn
with the ARN that you obtained in the output of theupdate-cluster-kafka-version
command.aws kafka describe-cluster-operation --cluster-operation-arn
ClusterOperationArn
The output of this
describe-cluster-operation
command looks like the following JSON example.{ "ClusterOperationInfo": { "ClientRequestId": "62cd41d2-1206-4ebf-85a8-dbb2ba0fe259", "ClusterArn": "arn:aws:kafka:us-east-1:012345678012:cluster/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2", "CreationTime": "2021-03-11T20:34:59.648000+00:00", "OperationArn": "arn:aws:kafka:us-east-1:012345678012:cluster-operation/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2/0123abcd-abcd-4f7f-1234-9876543210ef", "OperationState": "UPDATE_IN_PROGRESS", "OperationSteps": [ { "StepInfo": { "StepStatus": "IN_PROGRESS" }, "StepName": "INITIALIZE_UPDATE" }, { "StepInfo": { "StepStatus": "PENDING" }, "StepName": "UPDATE_APACHE_KAFKA_BINARIES" }, { "StepInfo": { "StepStatus": "PENDING" }, "StepName": "FINALIZE_UPDATE" } ], "OperationType": "UPDATE_CLUSTER_KAFKA_VERSION", "SourceClusterInfo": { "KafkaVersion": "2.4.1" }, "TargetClusterInfo": { "KafkaVersion": "2.6.1" } } }
If
OperationState
has the valueUPDATE_IN_PROGRESS
, wait a while, then run thedescribe-cluster-operation
command again. When the operation is complete, the value ofOperationState
becomesUPDATE_COMPLETE
. Because the time required for Amazon MSK to complete the operation varies, you might need to check repeatedly until the operation is complete.
Updating the Apache Kafka version using the API
Invoke the GetCompatibleKafkaVersions operation to get a list of the Apache Kafka versions to which you can update the cluster.
Invoke the UpdateClusterKafkaVersion operation to update the cluster to one of the compatible Apache Kafka versions.