Amazon MSK version support policy Updating the Apache Kafka version Best practices for version upgrades

Amazon MSK version support

This topic describes the Amazon MSK version support policy and the procedure for Updating the Apache Kafka version. If you're upgrading your Kafka version, follow the best practices outlined in Best practices for version upgrades.

Amazon MSK version support policy

This section describes the support policy for Amazon MSK supported Kafka versions.

All Kafka versions are supported until they reach their end of support date. For details on end of support dates, see Supported Apache Kafka versions. Upgrade your MSK cluster to the recommended Kafka version or higher version before the end of support date. For details on updating your Apache Kafka version, see Updating the Apache Kafka version. A cluster using a Kafka version after its end of support date is auto-upgraded to the recommended Kafka version.
MSK will phase out support for newly created clusters that use Kafka versions with published end of support dates.

Updating the Apache Kafka version

You can update an existing MSK cluster to a newer version of Apache Kafka. You can't update it to an older version. When you update the Apache Kafka version of an MSK cluster, also check your client-side software to make sure its version enables you to use the features of the cluster's new Apache Kafka version. Amazon MSK only updates the server software. It doesn't update your clients.

For information about how to make a cluster highly available during an update, see Build highly available clusters.

Important

You can't update the Apache Kafka version for an MSK cluster that exceeds the limits described in Right-size your cluster: Number of partitions per broker.

Updating the Apache Kafka version using the AWS Management Console

Open the Amazon MSK console at https://console.aws.amazon.com/msk/.
Choose the MSK cluster on which you want to update the Apache Kafka version.
On the Properties tab choose Upgrade in the Apache Kafka version section.

Updating the Apache Kafka version using the AWS CLI

Run the following command, replacing ClusterArn with the Amazon Resource Name (ARN) that you obtained when you created your cluster. If you don't have the ARN for your cluster, you can find it by listing all clusters. For more information, see Listing Amazon MSK clusters.
```
aws kafka get-compatible-kafka-versions --cluster-arn ClusterArn
```
The output of this command includes a list of the Apache Kafka versions to which you can update the cluster. It looks like the following example.
```
{
    "CompatibleKafkaVersions": [
        {
            "SourceVersion": "2.2.1",
            "TargetVersions": [
                "2.3.1",
                "2.4.1",
                "2.4.1.1",
                "2.5.1"
            ]
        }
    ]
}
```
Run the following command, replacing ClusterArn with the Amazon Resource Name (ARN) that you obtained when you created your cluster. If you don't have the ARN for your cluster, you can find it by listing all clusters. For more information, see Listing Amazon MSK clusters.

Replace Current-Cluster-Version with the current version of the cluster. For TargetVersion you can specify any of the target versions from the output of the previous command.

Important
Cluster versions aren't simple integers. To find the current version of the cluster, use the DescribeCluster operation or the describe-cluster AWS CLI command. An example version is KTVPDKIKX0DER.
```
aws kafka update-cluster-kafka-version --cluster-arn ClusterArn --current-version Current-Cluster-Version --target-kafka-version TargetVersion
```
The output of the previous command looks like the following JSON.
```
{
    
    "ClusterArn": "arn:aws:kafka:us-east-1:012345678012:cluster/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2",
    "ClusterOperationArn": "arn:aws:kafka:us-east-1:012345678012:cluster-operation/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2/0123abcd-abcd-4f7f-1234-9876543210ef"
}
```

To get the result of the update-cluster-kafka-version operation, run the following command, replacing ClusterOperationArn with the ARN that you obtained in the output of the update-cluster-kafka-version command.


aws kafka describe-cluster-operation --cluster-operation-arn ClusterOperationArn

The output of this describe-cluster-operation command looks like the following JSON example.


{
    "ClusterOperationInfo": {
        "ClientRequestId": "62cd41d2-1206-4ebf-85a8-dbb2ba0fe259",
        "ClusterArn": "arn:aws:kafka:us-east-1:012345678012:cluster/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2",
        "CreationTime": "2021-03-11T20:34:59.648000+00:00",
        "OperationArn": "arn:aws:kafka:us-east-1:012345678012:cluster-operation/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2/0123abcd-abcd-4f7f-1234-9876543210ef",
        "OperationState": "UPDATE_IN_PROGRESS",
        "OperationSteps": [
            {
                "StepInfo": {
                    "StepStatus": "IN_PROGRESS"
                },
                "StepName": "INITIALIZE_UPDATE"
            },
            {
                "StepInfo": {
                    "StepStatus": "PENDING"
                },
                "StepName": "UPDATE_APACHE_KAFKA_BINARIES"
            },
            {
                "StepInfo": {
                    "StepStatus": "PENDING"
                },
                "StepName": "FINALIZE_UPDATE"
            }
        ],
        "OperationType": "UPDATE_CLUSTER_KAFKA_VERSION",
        "SourceClusterInfo": {
            "KafkaVersion": "2.4.1"
        },
        "TargetClusterInfo": {
            "KafkaVersion": "2.6.1"
        }
    }
}

If OperationState has the value UPDATE_IN_PROGRESS, wait a while, then run the describe-cluster-operation command again. When the operation is complete, the value of OperationState becomes UPDATE_COMPLETE. Because the time required for Amazon MSK to complete the operation varies, you might need to check repeatedly until the operation is complete.

Updating the Apache Kafka version using the API

Invoke the GetCompatibleKafkaVersions operation to get a list of the Apache Kafka versions to which you can update the cluster.
Invoke the UpdateClusterKafkaVersion operation to update the cluster to one of the compatible Apache Kafka versions.

Best practices for version upgrades

To ensure client continuity during the rolling update that is performed as part of the Kafka version upgrade process, review the configuration of your clients and your Apache Kafka topics as follows:

Set the topic replication factor (RF) to a minimum value of 2 for two-AZ clusters and a minimum value of 3 for three-AZ clusters. An RF value of 2 can lead to offline partitions during patching.
Set minimum in-sync replicas (minISR) to a maximum value of RF - 1 to ensure the partition replica set can tolerate one replica being offline or under-replicated.
Configure clients to use multiple broker connection strings. Having multiple brokers in a client’s connection string allows for failover if a specific broker supporting client I/O begins to be patched. For information about how to get a connection string with multiple brokers, see Getting the bootstrap brokers for an Amazon MSK cluster.
We recommend that you upgrade connecting clients to the recommended version or above to benefit from the features available in the new version. Client upgrades are not subject to the end of life (EOL) dates of your MSK cluster's Kafka version, and do not need to be completed by the EOL date. Apache Kafka provides a bi-directional client compatibility policy that allows older clients to work with newer clusters and vice versa.
Kafka clients using versions 3.x.x are likely to come with the following defaults: acks=all and enable.idempotence=true. acks=all is different from the previous default of acks=1 and provides extra durability by ensuring that all in-sync replicas acknowledge the produce request. Similarly, the default for enable.idempotence was previously false. The change to enable.idempotence=true as the default lowers the likelihood of duplicate messages. These changes are considered best practice settings and may introduce a small amount of additional latency that's within normal performance parameters.
Use the recommended Kafka version when creating new MSK clusters. Using the recommended Kafka version allows you to benefit from the latest Kafka and MSK features.

Warning Javascript is disabled or is unavailable in your browser.

To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

Supported Apache Kafka versions

Troubleshooting