Managing the kube-proxy add-on - Amazon EKS

Managing the kube-proxy add-on

Kube-proxy maintains network rules on each Amazon EC2 node. It enables network communication to your pods. Kube-proxy is not deployed to Fargate nodes. For more information, see kube-proxy in the Kubernetes documentation. There are two types of the kube-proxy image available for each Kubernetes version:

  • Default – This type is deployed by default with new clusters. This is the only type that you can use with the Amazon EKS add-on for version 1.22 and earlier clusters.

  • Minimal – Unlike the default type, this image type is based on a minimal base image maintained by Amazon EKS Distro, which contains minimal packages and doesn't have shells. For more information, see Amazon EKS Distro. This type is available as a self-managed add-on. It is also available as an Amazon EKS add-on for version 1.23 and later clusters, but isn't available as an Amazon EKS add-on for version 1.22 and earlier clusters. If you choose to install this type as a self-managed add-on, for version 1.22 and earlier clusters, complete the steps in Updating the kube-proxy self-managed add-on, specifying the latest version for the minimal type in the following table. This type is added by default to any new 1.23 and later clusters. If you update an existing cluster to version 1.23, you can add the Amazon EKS add-on to your cluster, or update your existing Amazon EKS add-on to this type. Even though the v1.23.7-eksbuild.1 version doesn't include minimal in the version (as previous versions do), the image is still based on the minimal base image.

Latest available kube-proxy version for each Amazon EKS cluster version
Image type 1.23 1.22 1.21 1.20 1.19
kube-proxy (default type) Not available v1.22.11-eksbuild.2 v1.21.14-eksbuild.2 v1.20.15-eksbuild.2 v1.19.16-eksbuild.2
kube-proxy (minimal type) v1.23.7-eksbuild.1 v1.22.11-minimal-eksbuild.2 v1.21.14-minimal-eksbuild.2 v1.20.15-minimal-eksbuild.3 v1.19.16-minimal-eksbuild.3

If you created a 1.18 or later cluster using the AWS Management Console, then Amazon EKS installed the plugin for you as an Amazon EKS add-on. If you originally created a 1.17 or earlier cluster using any tool, or you created a 1.18 or later cluster using any tool other than the AWS Management Console, then Amazon EKS installed the plugin as a self-managed add-on for you. You can migrate the self-managed add-on to the Amazon EKS add-on using the procedure in Adding the kube-proxy Amazon EKS add-on. If you have a cluster that you've already added the kube-proxy add-on to, you can manage it using the procedures in the Updating the kube-proxy Amazon EKS add-on and Removing the kube-proxy Amazon EKS add-on sections. For more information about Amazon EKS add-ons, see Amazon EKS add-ons.

If you haven't added the kube-proxy Amazon EKS add-on, the kube-proxy self-managed add-on is still running on your cluster. You can manually update the kube-proxy self-managed add-on using the procedure in Updating the kube-proxy self-managed add-on.

Amazon EKS has the same compatibility and skew policy as Kubernetes for kube-proxy. Kube-proxy must meet the following requirements:

  • It must be the same minor version as kubelet on the node.

  • It must not be newer than the version of your cluster's control plane.

  • It's version can't be earlier than two minor versions from your control plane. For example, if your control plane is running Kubernetes 1.23, then the kube-proxy minor version can't be earlier than 1.21.

Prerequisites

Adding the kube-proxy Amazon EKS add-on

You can add the kube-proxy Amazon EKS add-on to your cluster using eksctl, the AWS Management Console, or the AWS CLI.

Important

Before adding the kube-proxy Amazon EKS add-on, confirm that you do not self-manage any settings that Amazon EKS will start managing. To determine which settings Amazon EKS manages, see Amazon EKS add-on configuration.

eksctl

To add the kube-proxy Amazon EKS add-on using eksctl

Replace my-cluster with the name of your cluster and then run the following command. Eksctl adds the latest version of the add-on that's available for your cluster version.

eksctl create addon --name kube-proxy --cluster my-cluster --force

If you remove the --force option and any of the Amazon EKS add-on settings conflict with your existing settings, then adding the Amazon EKS add-on fails, and you receive an error message to help you resolve the conflict. Before specifying this option, make sure that the Amazon EKS add-on doesn't manage settings that you need to manage, because those settings are overwritten with this option. For more information about Amazon EKS add-on configuration management, see Amazon EKS add-on configuration.

AWS Management Console

To add the kube-proxy Amazon EKS add-on using the AWS Management Console

  1. Open the Amazon EKS console at https://console.aws.amazon.com/eks/home#/clusters.

  2. In the left navigation pane, select Clusters, and then select the name of the cluster that you want to configure the kube-proxy Amazon EKS add-on for.

  3. Choose the Add-ons tab.

  4. Select Add new.

    1. Select kube-proxy for Name.

    2. Select the Version you'd like to use. Select the version marked as Latest.

    3. If you select Override existing configuration for this add-on on the cluster., then any setting for the existing add-on can be overwritten with the Amazon EKS add-on's settings. If you don't enable this option and any of the Amazon EKS add-on settings conflict with your existing settings, then adding the Amazon EKS add-on fails, and you receive an error message to help you resolve the conflict. Before selecting this option, make sure that the Amazon EKS add-on doesn't manage settings that you need to manage. For more information about Amazon EKS add-on configuration management, see Amazon EKS add-on configuration.

    4. Select Add.

AWS CLI

To add the kube-proxy Amazon EKS add-on using the AWS CLI

Create the add-on. Replace my-cluster with the name of your cluster and 1.23.7-minimal-eksbuild.1 with the latest version for your cluster from the latest versions table.

aws eks create-addon --cluster-name my-cluster --addon-name kube-proxy --addon-version v1.23.7-minimal-eksbuild.1 --resolve-conflicts OVERWRITE

If you remove the --resolve-conflicts OVERWRITE option and any of the Amazon EKS add-on settings conflict with your existing settings, then creating the add-on fails, and you receive an error message to help you resolve the conflict. Before specifying this option, make sure that the Amazon EKS add-on doesn't manage settings that you need to manage, because those settings are overwritten with this option. For more information about Amazon EKS add-on configuration management, see Amazon EKS add-on configuration.

Updating the kube-proxy Amazon EKS add-on

This procedure is for updating the kube-proxy Amazon EKS add-on. If you haven't added the kube-proxy Amazon EKS add-on, complete the procedure in Updating the kube-proxy self-managed add-on instead. Amazon EKS does not automatically update kube-proxy on your cluster when new versions are released or after you update your cluster to a new Kubernetes minor version. To update kube-proxy on an existing cluster, you must initiate the update and then Amazon EKS updates the add-on for you.

Important

Update your cluster and nodes to a new Kubernetes minor version before updating kube-proxy to the same minor version as your updated cluster's minor version.

eksctl

To update the kube-proxy Amazon EKS add-on using eksctl

  1. Check the current version of your kube-proxy Amazon EKS add-on. Replace my-cluster with your cluster name.

    eksctl get addon --name kube-proxy --cluster my-cluster

    The example output is as follows.

    NAME VERSION STATUS ISSUES IAMROLE UPDATE AVAILABLE kube-proxy v1.22.11-eksbuild.2 ACTIVE 0 v1.23.7-minimal-eksbuild.1
  2. Update the add-on to the version returned under UPDATE AVAILABLE in the output of the previous step.

    eksctl update addon \ --name kube-proxy \ --version v1.23.7-minimal-eksbuild.1 \ --cluster my-cluster \ --force

    If you remove the --force option and any of the Amazon EKS add-on settings conflict with your existing settings, then updating the add-on fails, and you receive an error message to help you resolve the conflict. Before specifying this option, make sure that the Amazon EKS add-on doesn't manage settings that you need to manage, because those settings are overwritten with this option. For more information about Amazon EKS add-on configuration management, see Amazon EKS add-on configuration.

AWS Management Console

To update the kube-proxy Amazon EKS add-on using the AWS Management Console

  1. Open the Amazon EKS console at https://console.aws.amazon.com/eks/home#/clusters.

  2. In the left navigation pane, select Clusters, and then select the name of the cluster that you want to update the kube-proxy Amazon EKS add-on for.

  3. Choose the Add-ons tab.

  4. Select the box in the top right of the kube-proxy box and then choose Edit.

    1. Select the Version marked as Latest.

    2. If you select Override existing configuration for this add-on on the cluster., then any setting for the existing add-on can be overwritten with the Amazon EKS add-on's settings. If you don't enable this option and any of the Amazon EKS add-on settings conflict with your existing settings, then updating the add-on to an Amazon EKS add-on fail, and you receive an error message to help you resolve the conflict. Before selecting this option, make sure that the Amazon EKS add-on doesn't manage settings that you need to manage. For more information about Amazon EKS add-on configuration management, see Amazon EKS add-on configuration.

    3. Select Update.

AWS CLI

To update the kube-proxy Amazon EKS add-on using the AWS CLI

  1. Check the current version of your kube-proxy Amazon EKS add-on. Replace my-cluster with your cluster name.

    aws eks describe-addon --cluster-name my-cluster --addon-name kube-proxy --query "addon.addonVersion" --output text

    The example output is as follows.

    v1.22.11-eksbuild.2
  2. Determine which versions of the kube-proxy add-on are available for your cluster version. Replace 1.23 with your cluster's version.

    aws eks describe-addon-versions --addon-name kube-proxy --kubernetes-version 1.23 \ --query "addons[].addonVersions[].[addonVersion, compatibilities[].defaultVersion]" --output text

    The example output is as follows.

    v1.23.7-minimal-eksbuild.1 False v1.22.11-eksbuild.2 True ...

    The version with True underneath is the default version deployed with new clusters with the Kubernetes version that you specified.

  3. Update the add-on to the latest version returned in the output of the previous step.

    aws eks update-addon --cluster-name my-cluster --addon-name kube-proxy --addon-version v1.23.7-minimal-eksbuild.1 --resolve-conflicts OVERWRITE

    If you remove the --resolve-conflicts OVERWRITE option and any of the Amazon EKS add-on settings conflict with your existing settings, then updating the add-on fails, and you receive an error message to help you resolve the conflict. Before specifying this option, make sure that the Amazon EKS add-on doesn't manage settings that you need to manage, because those settings are overwritten with this option. For more information about Amazon EKS add-on configuration management, see Amazon EKS add-on configuration.

Removing the kube-proxy Amazon EKS add-on

You have two options when removing an Amazon EKS add-on:

  • Preserve the add-on's software on your cluster – This option removes Amazon EKS management of any settings and the ability for Amazon EKS to notify you of updates and automatically update the Amazon EKS add-on after you initiate an update, but preserves the add-on's software on your cluster. This option makes the add-on a self-managed add-on, rather than an Amazon EKS add-on. There is no downtime for the add-on.

  • Removing the add-on software entirely from your cluster – You should only remove the Amazon EKS add-on from your cluster if there are no resources on your cluster are dependent on the functionality that the add-on provides. After removing the Amazon EKS add-on, you can add it again if you want to.

If the add-on has an IAM account associated with it, the IAM account is not removed.

You can use eksctl, the AWS Management Console or the AWS CLI to remove the kube-proxy Amazon EKS add-on from your cluster.

eksctl

To remove the kube-proxy Amazon EKS add-on using eksctl

Replace my-cluster with the name of your cluster and then run the following command. Removing --preserve removes the add-on software from your cluster.

eksctl delete addon --cluster my-cluster --name kube-proxy --preserve
AWS Management Console

To remove the kube-proxy Amazon EKS add-on using the AWS Management Console

  1. Open the Amazon EKS console at https://console.aws.amazon.com/eks/home#/clusters.

  2. In the left navigation pane, select Clusters, and then select the name of the cluster that you want to remove the kube-proxy Amazon EKS add-on for.

  3. Choose the Add-ons tab.

  4. Select the check box in the top right of the kube-proxy box and then choose Remove. Type kube-proxy and then select Remove.

AWS CLI

To remove the kube-proxy Amazon EKS add-on using the AWS CLI

Replace my-cluster with the name of your cluster and then run the following command. Removing --preserve removes the add-on software from your cluster.

aws eks delete-addon --cluster-name my-cluster --addon-name kube-proxy --preserve

Updating the kube-proxy self-managed add-on

If you have a cluster that you have not added the kube-proxy Amazon EKS add-on to, complete the following steps to update the self-managed add-on. If you've added the kube-proxy Amazon EKS add-on, complete the procedure in Updating the kube-proxy Amazon EKS add-on instead.

Important

Update your cluster and nodes to a new Kubernetes minor version before updating kube-proxy to the same minor version as your updated cluster's minor version.

To update the kube-proxy self-managed add-on using kubectl

  1. Check the current version of your kube-proxy deployment.

    kubectl get daemonset kube-proxy --namespace kube-system -o=jsonpath='{$.spec.template.spec.containers[:1].image}'

    The example output is as follows.

    602401143452.dkr.ecr.region-code.amazonaws.com/eks/kube-proxy:v1.22.11-eksbuild.2
  2. Update the kube-proxy add-on by replacing 602401143452 and region-code with the values from your output. Replace 1.22.11-eksbuild.2 with the kube-proxy version listed in the Latest available kube-proxy version for each Amazon EKS cluster version table for your cluster version. You can specify a version number for the default or minimal image type.

    kubectl set image daemonset.apps/kube-proxy -n kube-system kube-proxy=602401143452.dkr.ecr.region-code.amazonaws.com/eks/kube-proxy:v1.23.7-minimal-eksbuild.1
  3. If you're using x86 and Arm nodes in the same cluster and your cluster was deployed before August 17, 2020. Then, edit your kube-proxy manifest to include a node selector for multiple hardware architectures with the following command. This is a one-time operation. After you've added the selector to your manifest, you don't need to add it each time you update. If your cluster was deployed on or after August 17, 2020, then kube-proxy is already multi-architecture capable.

    kubectl edit -n kube-system daemonset/kube-proxy

    Add the following node selector to the file in the editor and then save the file. For an example of where to include this text in the editor, see the CNI manifest file on GitHub. This enables Kubernetes to pull the correct hardware image based on the node's hardware architecture.

    - key: "kubernetes.io/arch" operator: In values: - amd64 - arm64
  4. If your cluster was originally created with Kubernetes version 1.14 or later, then you can skip this step because kube-proxy already includes this Affinity Rule. If you originally created an Amazon EKS cluster with Kubernetes version 1.13 or earlier and intend to use Fargate nodes, then edit your kube-proxy manifest to include a NodeAffinity rule to prevent kube-proxy pods from scheduling on Fargate nodes. This is a one-time edit. Once you've added the Affinity Rule to your manifest, you don't need to add it each time you upgrade your cluster. Edit your kube-proxy DaemonSet.

    kubectl edit -n kube-system daemonset/kube-proxy

    Add the following Affinity Rule to the DaemonSet spec section of the file in the editor and then save the file. For an example of where to include this text in the editor, see the CNI manifest file on GitHub.

    - key: eks.amazonaws.com/compute-type operator: NotIn values: - fargate