Review release notes for Kubernetes versions on extended support - Amazon EKS
Kubernetes 1.27Kubernetes 1.26Kubernetes 1.25Kubernetes 1.24Kubernetes 1.23

Review release notes for Kubernetes versions on extended support

This topic gives important changes to be aware of for each Kubernetes version in extended support. When upgrading, carefully review the changes that have occurred between the old and new versions for your cluster.

Kubernetes 1.27

Kubernetes 1.27 is now available in Amazon EKS. For more information about Kubernetes 1.27, see the official release announcement.

Important

  • The support for the alpha seccomp annotations seccomp.security.alpha.kubernetes.io/pod and container.seccomp.security.alpha.kubernetes.io annotations was removed. The alpha seccomp annotations was deprecated in 1.19, and with their removal in 1.27, seccomp fields will no longer auto-populate for Pods with seccomp annotations. Instead, use the securityContext.seccompProfile field for Pods or containers to configure seccomp profiles. To check whether you are using the deprecated alpha seccomp annotations in your cluster, run the following command:

    kubectl get pods --all-namespaces -o json | grep -E 'seccomp.security.alpha.kubernetes.io/pod|container.seccomp.security.alpha.kubernetes.io'

  • The --container-runtime command line argument for the kubelet was removed. The default container runtime for Amazon EKS has been containerd since 1.24, which eliminates the need to specify the container runtime. From 1.27 onwards, Amazon EKS will ignore the --container-runtime argument passed to any bootstrap scripts. It is important that you don’t pass this argument to --kubelet-extra-args in order to prevent errors during the node bootstrap process. You must remove the --container-runtime argument from all of your node creation workflows and build scripts.

  • The kubelet in Kubernetes 1.27 increased the default kubeAPIQPS to 50 and kubeAPIBurst to 100. These enhancements allow the kubelet to handle a higher volume of API queries, improving response times and performance. When the demands for Pods increase, due to scaling requirements, the revised defaults ensure that the kubelet can efficiently manage the increased workload. As a result, Pod launches are quicker and cluster operations are more effective.

  • You can use more fine grained Pod topology to spread policies such as minDomain. This parameter gives you the ability to specify the minimum number of domains your Pods should be spread across. nodeAffinityPolicy and nodeTaintPolicy allow for an extra level of granularity in governing Pod distribution. This is in accordance to node affinities, taints, and the matchLabelKeys field in the topologySpreadConstraints of your Pod’s specification. This permits the selection of Pods for spreading calculations following a rolling upgrade.

  • Kubernetes 1.27 promoted to beta a new policy mechanism for StatefulSets that controls the lifetime of their PersistentVolumeClaims(PVCs). The new PVC retention policy lets you specify if the PVCs generated from the StatefulSet spec template will be automatically deleted or retained when the StatefulSet is deleted or replicas in the StatefulSet are scaled down.

  • The goaway-chance option in the Kubernetes API server helps prevent HTTP/2 client connections from being stuck on a single API server instance, by randomly closing a connection. When the connection is closed, the client will try to reconnect, and will likely land on a different API server as a result of load balancing. Amazon EKS version 1.27 has enabled goaway-chance flag. If your workload running on Amazon EKS cluster uses a client that is not compatible with HTTP GOAWAY, we recommend that you update your client to handle GOAWAY by reconnecting on connection termination.

For the complete Kubernetes 1.27 changelog, see https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.27.md#changelog-since-v1260.

Kubernetes 1.26

Kubernetes 1.26 is now available in Amazon EKS. For more information about Kubernetes 1.26, see the official release announcement.

Important

Kubernetes 1.26 no longer supports CRI v1alpha2. This results in the kubelet no longer registering the node if the container runtime doesn’t support CRI v1. This also means that Kubernetes 1.26 doesn’t support containerd minor version 1.5 and earlier. If you’re using containerd, you need to upgrade to containerd version 1.6.0 or later before you upgrade any nodes to Kubernetes 1.26. You also need to upgrade any other container runtimes that only support the v1alpha2. For more information, defer to the container runtime vendor. By default, Amazon Linux and Bottlerocket AMIs include containerd version 1.6.6.

  • Before you upgrade to Kubernetes 1.26, upgrade your Amazon VPC CNI plugin for Kubernetes to version 1.12 or later. If you don’t upgrade to Amazon VPC CNI plugin for Kubernetes version 1.12 or later, the Amazon VPC CNI plugin for Kubernetes will crash. For more information, see Amazon VPC CNI.

  • The goaway-chance option in the Kubernetes API server helps prevent HTTP/2 client connections from being stuck on a single API server instance, by randomly closing a connection. When the connection is closed, the client will try to reconnect, and will likely land on a different API server as a result of load balancing. Amazon EKS version 1.26 has enabled goaway-chance flag. If your workload running on Amazon EKS cluster uses a client that is not compatible with HTTP GOAWAY, we recommend that you update your client to handle GOAWAY by reconnecting on connection termination.

For the complete Kubernetes 1.26 changelog, see https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.26.md#changelog-since-v1250.

Kubernetes 1.25

Kubernetes 1.25 is now available in Amazon EKS. For more information about Kubernetes 1.25, see the official release announcement.

Important

  • Amazon EC2 P2 instances aren’t supported on Amazon EKS because they require NVIDIA driver version 470 or earlier.

  • PodSecurityPolicy (PSP) is removed in Kubernetes 1.25. PSPs are replaced with Pod Security Admission (PSA) and Pod Security Standards (PSS). PSA is a built-in admission controller that implements the security controls outlined in the PSS. PSA and PSS are graduated to stable in Kubernetes 1.25 and are enabled in Amazon EKS by default. If you have PSPs in your cluster, make sure to migrate from PSP to the built-in Kubernetes PSS or to a policy-as-code solution before upgrading your cluster to version 1.25. If you don’t migrate from PSP, you might encounter interruptions to your workloads. For more information, see the Migrate from legacy pod security policies (PSP)Migrate from legacy pod security policies (PSP).

  • Kubernetes version 1.25 contains changes that alter the behavior of an existing feature known as API Priority and Fairness (APF). APF serves to shield the API server from potential overload during periods of heightened request volumes. It does this by placing restrictions on the number of concurrent requests that can be processed at any given time. This is achieved through the application of distinct priority levels and limits to requests originating from various workloads or users. This approach ensures that critical applications or high-priority requests receive preferential treatment, while simultaneously preventing lower priority requests from overwhelming the API server. For more information, see API Priority and Fairness in the Kubernetes documentation or API Priority and Fairness in the EKS Best Practices Guide.

    These updates were introduced in PR #10352 and PR #118601. Previously, APF treated all types of requests uniformly, with each request consuming a single unit of the concurrent request limit. The APF behavior change assigns higher units of concurrency to LIST requests due to the exceptionally heavy burden put on the API server by these requests. The API server estimates the number of objects that will be returned by a LIST request. It assigns a unit of concurrency that is proportional to the number of objects returned.

    Upon upgrading to Amazon EKS version 1.25 or higher, this updated behavior might cause workloads with heavy LIST requests (that previously functioned without issue) to encounter rate limiting. This would be indicated by an HTTP 429 response code. To avoid potential workload disruption due to LIST requests being rate limited, we strongly encourage you to restructure your workloads to reduce the rate of these requests. Alternatively, you can address this issue by adjusting the APF settings to allocate more capacity for essential requests while reducing the capacity allocated to non-essential ones. For more information about these mitigation techniques, see Preventing Dropped Requests in the EKS Best Practices Guide.

  • Amazon EKS 1.25 includes enhancements to cluster authentication that contain updated YAML libraries. If a YAML value in the aws-auth ConfigMap found in the kube-system namespace starts with a macro, where the first character is a curly brace, you should add quotation marks ("` "{ }`). This is required to ensure that aws-iam-authenticator version v0.6.3 accurately parses the aws-auth ConfigMap in Amazon EKS 1.25.

  • The beta API version (discovery.k8s.io/v1beta1) of EndpointSlice was deprecated in Kubernetes 1.21 and is no longer served as of Kubernetes 1.25. This API has been updated to discovery.k8s.io/v1. For more information, see EndpointSlice in the Kubernetes documentation. The AWS Load Balancer Controller v2.4.6 and earlier used the v1beta1 endpoint to communicate with EndpointSlices. If you’re using the EndpointSlices configuration for the AWS Load Balancer Controller, you must upgrade to AWS Load Balancer Controller v2.4.7`before upgrading your Amazon EKS cluster to `1.25. If you upgrade to 1.25 while using the EndpointSlices configuration for the AWS Load Balancer Controller, the controller will crash and result in interruptions to your workloads. To upgrade the controller, see Route internet traffic with AWS Load Balancer Controller.

  • The beta API version (autoscaling/v2beta1) of HorizontalPodAutoscaler is no longer served as of Kubernetes 1.25. This API was deprecated in version 1.23. Migrate manifests and API clients to use the autoscaling/v2 HorizontalPodAutoscaler API version. For more information, see the Kubernetes documentation.

  • SeccompDefault is promoted to beta in Kubernetes 1.25. By setting the --seccomp-default flag when you configure kubelet, the container runtime uses its RuntimeDefaultseccomp profile, rather than the unconfined (seccomp disabled) mode. The default profiles provide a strong set of security defaults, while preserving the functionality of the workload. Although this flag is available, Amazon EKS doesn’t enable this flag by default, so Amazon EKS behavior is effectively unchanged. If you want to, you can start enabling this on your nodes. For more details, see the tutorial Restrict a Container’s Syscalls with seccomp in the Kubernetes documentation.

  • Support for the Container Runtime Interface (CRI) for Docker (also known as dockershim) was removed from Kubernetes 1.24 and later. The only container runtime in Amazon EKS official AMIs for Kubernetes 1.24 and later clusters is containerd. Before upgrading to Amazon EKS 1.24 or later, remove any reference to bootstrap script flags that aren’t supported anymore. For more information, see Migrate from dockershim to containerd.

  • The support for wildcard queries was deprecated in CoreDNS 1.8.7 and removed in CoreDNS 1.9. This was done as a security measure. Wildcard queries no longer work and return NXDOMAIN instead of an IP address.

  • The goaway-chance option in the Kubernetes API server helps prevent HTTP/2 client connections from being stuck on a single API server instance, by randomly closing a connection. When the connection is closed, the client will try to reconnect, and will likely land on a different API server as a result of load balancing. Amazon EKS version 1.25 has enabled goaway-chance flag. If your workload running on Amazon EKS cluster uses a client that is not compatible with HTTP GOAWAY, we recommend that you update your client to handle GOAWAY by reconnecting on connection termination.

For the complete Kubernetes 1.25 changelog, see https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.25.md#changelog-since-v1240.

Kubernetes 1.24

Kubernetes 1.24 is now available in Amazon EKS. For more information about Kubernetes 1.24, see the official release announcement.

Important

  • Starting with Kubernetes 1.24, new beta APIs aren’t enabled in clusters by default. By default, existing beta APIs and new versions of existing beta APIs continue to be enabled. Amazon EKS follows the same behavior as upstream Kubernetes 1.24. The feature gates that control new features for both new and existing API operations are enabled by default. This is in alignment with upstream Kubernetes. For more information, see KEP-3136: Beta APIs Are Off by Default on GitHub.

  • Support for Container Runtime Interface (CRI) for Docker (also known as dockershim) is removed from Kubernetes 1.24. Amazon EKS official AMIs have containerd as the only runtime. Before moving to Amazon EKS 1.24 or higher, you must remove any reference to bootstrap script flags that aren’t supported anymore. You must also make sure that IP forwarding is enabled for your worker nodes. For more information, see Migrate from dockershim to containerd.

  • If you already have Fluentd configured for Container Insights, then you must migrate Fluentd to Fluent Bit before updating your cluster. The Fluentd parsers are configured to only parse log messages in JSON format. Unlike dockerd, the containerd container runtime has log messages that aren’t in JSON format. If you don’t migrate to Fluent Bit, some of the configured Fluentd’s parsers will generate a massive amount of errors inside the Fluentd container. For more information on migrating, see Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs.

  • In Kubernetes 1.23 and earlier, kubelet serving certificates with unverifiable IP and DNS Subject Alternative Names (SANs) are automatically issued with unverifiable SANs. These unverifiable SANs are omitted from the provisioned certificate. In version 1.24 and later clusters, kubelet serving certificates aren’t issued if any SAN can’t be verified. This prevents kubectl exec and kubectl logs commands from working. For more information, see Certificate signing considerations before upgrading your cluster to Kubernetes 1.24.

  • When upgrading an Amazon EKS 1.23 cluster that uses Fluent Bit, you must make sure that it’s running k8s/1.3.12 or later. You can do this by reapplying the latest applicable Fluent Bit YAML file from GitHub. For more information, see Setting up Fluent Bit in the Amazon CloudWatch User Guide.

  • You can use Topology Aware Hints to indicate your preference for keeping traffic in zone when cluster worker nodes are deployed across multiple availability zones. Routing traffic within a zone can help reduce costs and improve network performance. By default, Topology Aware Hints are enabled in Amazon EKS 1.24. For more information, see Topology Aware Hints in the Kubernetes documentation.

  • The PodSecurityPolicy (PSP) is scheduled for removal in Kubernetes 1.25. PSPs are being replaced with Pod Security Admission (PSA). PSA is a built-in admission controller that uses the security controls that are outlined in the Pod Security Standards (PSS). PSA and PSS are both beta features and are enabled in Amazon EKS by default. To address the removal of PSP in version 1.25, we recommend that you implement PSS in Amazon EKS. For more information, see Implementing Pod Security Standards in Amazon EKS on the AWS blog.

  • The client.authentication.k8s.io/v1alpha1 ExecCredential is removed in Kubernetes 1.24. The ExecCredential API was generally available in Kubernetes 1.22. If you use a client-go credential plugin that relies on the v1alpha1 API, contact the distributor of your plugin on how to migrate to the v1 API.

  • For Kubernetes 1.24, we contributed a feature to the upstream Cluster Autoscaler project that simplifies scaling Amazon EKS managed node groups to and from zero nodes. Previously, for the Cluster Autoscaler to understand the resources, labels, and taints of a managed node group that was scaled to zero nodes, you needed to tag the underlying Amazon EC2 Auto Scaling group with the details of the nodes that it was responsible for. Now, when there are no running nodes in the managed node group, the Cluster Autoscaler calls the Amazon EKS DescribeNodegroup API operation. This API operation provides the information that the Cluster Autoscaler requires of the managed node group’s resources, labels, and taints. This feature requires that you add the eks:DescribeNodegroup permission to the Cluster Autoscaler service account IAM policy. When the value of a Cluster Autoscaler tag on the Auto Scaling group powering an Amazon EKS managed node group conflicts with the node group itself, the Cluster Autoscaler prefers the value of the Auto Scaling group tag. This is so that you can override values as needed. For more information, see Cluster Autoscaler.

  • If you intend to use Inferentia or Trainium instance types with Amazon EKS 1.24, you must upgrade to the AWS Neuron device plugin version 1.9.3.0 or later. For more information, see Neuron K8 release [1.9.3.0] in the AWS Neuron Documentation.

  • Containerd has IPv6 enabled for Pods, by default. It applies node kernel settings to Pod network namespaces. Because of this, containers in a Pod bind to both IPv4 (127.0.0.1) and IPv6 (::1) loopback addresses. IPv6 is the default protocol for communication. Before updating your cluster to version 1.24, we recommend that you test your multi-container Pods. Modify apps so that they can bind to all IP addresses on loopback interfaces. The majority of libraries enable IPv6 binding, which is backward compatible with IPv4. When it’s not possible to modify your application code, you have two options:

    • Run an init container and set disable ipv6 to true (sysctl -w net.ipv6.conf.all.disable_ipv6=1).

    • Configure a mutating admission webhook to inject an init container alongside your application Pods.

    If you need to block IPv6 for all Pods across all nodes, you might have to disable IPv6 on your instances.

  • The goaway-chance option in the Kubernetes API server helps prevent HTTP/2 client connections from being stuck on a single API server instance, by randomly closing a connection. When the connection is closed, the client will try to reconnect, and will likely land on a different API server as a result of load balancing. Amazon EKS version 1.24 has enabled goaway-chance flag. If your workload running on Amazon EKS cluster uses a client that is not compatible with HTTP GOAWAY, we recommend that you update your client to handle GOAWAY by reconnecting on connection termination.

For the complete Kubernetes 1.24 changelog, see https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#changelog-since-v1230.

Kubernetes 1.23

Kubernetes 1.23 is now available in Amazon EKS. For more information about Kubernetes 1.23, see the official release announcement.

Important

  • The Kubernetes in-tree to container storage interface (CSI) volume migration feature is enabled. This feature enables the replacement of existing Kubernetes in-tree storage plugins for Amazon EBS with a corresponding Amazon EBS CSI driver. For more information, see Kubernetes 1.17 Feature: Kubernetes In-Tree to CSI Volume Migration Moves to Beta on the Kubernetes blog.

    The feature translates in-tree APIs to equivalent CSI APIs and delegates operations to a replacement CSI driver. With this feature, if you use existing StorageClass, PersistentVolume, and PersistentVolumeClaim objects that belong to these workloads, there likely won’t be any noticeable change. The feature enables Kubernetes to delegate all storage management operations from the in-tree plugin to the CSI driver. If you use Amazon EBS volumes in an existing cluster, install the Amazon EBS CSI driver in your cluster before you update your cluster to version 1.23. If you don’t install the driver before updating an existing cluster, interruptions to your workloads might occur. If you plan to deploy workloads that use Amazon EBS volumes in a new 1.23 cluster, install the Amazon EBS CSI driver in your cluster before deploying the workloads your cluster. For instructions on how to install the Amazon EBS CSI driver on your cluster, see Store Kubernetes volumes with Amazon EBS. For frequently asked questions about the migration feature, see Amazon EBS CSI migration frequently asked questions.

  • Extended Support for Amazon EKS optimized Windows AMIs that are published by AWS isn’t available for Kubernetes version 1.23 but is available for Kubernetes version 1.24 and higher.

  • Kubernetes stopped supporting dockershim in version 1.20 and removed dockershim in version 1.24. For more information, see Kubernetes is Moving on From Dockershim: Commitments and Next Steps in the Kubernetes blog. Amazon EKS will end support for dockershim starting in Amazon EKS version 1.24. Starting with Amazon EKS version 1.24, Amazon EKS official AMIs will have containerd as the only runtime.

    Even though Amazon EKS version 1.23 continues to support dockershim, we recommend that you start testing your applications now to identify and remove any Docker dependencies. This way, you are prepared to update your cluster to version 1.24. For more information about dockershim removal, see Migrate from dockershim to containerd.

  • Kubernetes graduated IPv4/IPv6 dual-stack networking for Pods, services, and nodes to general availability. However, Amazon EKS and the Amazon VPC CNI plugin for Kubernetes don’t support dual-stack networking. Your clusters can assign IPv4 or IPv6 addresses to Pods and services, but can’t assign both address types.

  • Kubernetes graduated the Pod Security Admission (PSA) feature to beta. The feature is enabled by default. For more information, see Pod Security Admission in the Kubernetes documentation. PSA replaces the Pod Security Policy (PSP) admission controller. The PSP admission controller isn’t supported and is scheduled for removal in Kubernetes version 1.25.

    The PSP admission controller enforces Pod security standards on Pods in a namespace based on specific namespace labels that set the enforcement level. For more information, see Pod Security Standards (PSS) and Pod Security Admission (PSA) in the Amazon EKS best practices guide.

  • The kube-proxy image deployed with clusters is now the minimal base image maintained by Amazon EKS Distro (EKS-D). The image contains minimal packages and doesn’t have shells or package managers.

  • Kubernetes graduated ephemeral containers to beta. Ephemeral containers are temporary containers that run in the same namespace as an existing Pod. You can use them to observe the state of Pods and containers for troubleshooting and debugging purposes. This is especially useful for interactive troubleshooting when kubectl exec is insufficient because either a container has crashed or a container image doesn’t include debugging utilities. An example of a container that includes a debugging utility is distroless images. For more information, see Debugging with an ephemeral debug container in the Kubernetes documentation.

  • Kubernetes graduated the HorizontalPodAutoscaler autoscaling/v2 stable API to general availability. The HorizontalPodAutoscaler autoscaling/v2beta2 API is deprecated. It will be unavailable in 1.26.

  • The goaway-chance option in the Kubernetes API server helps prevent HTTP/2 client connections from being stuck on a single API server instance, by randomly closing a connection. When the connection is closed, the client will try to reconnect, and will likely land on a different API server as a result of load balancing. Amazon EKS version 1.23 has enabled goaway-chance flag. If your workload running on Amazon EKS cluster uses a client that is not compatible with HTTP GOAWAY, we recommend that you update your client to handle GOAWAY by reconnecting on connection termination.

For the complete Kubernetes 1.23 changelog, see https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#changelog-since-v1220.