Running kube-proxy in IPVS Mode

EKS in IP Virtual Server (IPVS) mode solves the network latency issue often seen when running large clusters with over 1,000 services with kube-proxy running in legacy iptables mode. This performance issue is the result of sequential processing of iptables packet filtering rules for each packet. This latency issue has been addressed in nftables, the successor to iptables. However, as of the time of this writing, kube-proxy is still under development to make use of nftables. To get around this issue, you can configure your cluster to run kube-proxy in IPVS mode.

Overview

IPVS, which has been GA since Kubernetes version 1.11, uses hash tables rather than linear searching to process packets, providing efficiency for clusters with thousands of nodes and services. IPVS was designed for load balancing, making it a suitable solution for Kubernetes networking performance issues.

IPVS offers several options for distributing traffic to backend pods. Detailed information for each option can be found in the official Kubernetes documentation, but a simple list is shown below. Round Robin and Least Connections are among the most popular choices for IPVS load balancing options in Kubernetes.

- rr (Round Robin)
- wrr (Weighted Round Robin)
- lc (Least Connections)
- wlc (Weighted Least Connections)
- lblc (Locality Based Least Connections)
- lblcr (Locality Based Least Connections with Replication)
- sh (Source Hashing)
- dh (Destination Hashing)
- sed (Shortest Expected Delay)
- nq (Never Queue)

Implementation

Only a few steps are required to enable IPVS in your EKS cluster. The first thing you need to do is ensure your EKS worker node images have the Linux Virtual Server administration ipvsadm package installed. To install this package on a Fedora based image, such as Amazon Linux 2023, you can run the following command on the worker node instance.

sudo dnf install -y ipvsadm

On a Debian based image, such as Ubuntu, the installation command would look like this.

sudo apt-get install ipvsadm

Next, you need to load the kernel modules for the IPVS configuration options listed above. We recommend writing these modules to a file inside of the /etc/modules-load.d/ directory so that they survive a reboot.

sudo sh -c 'cat << EOF > /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_lc
ip_vs_wlc
ip_vs_lblc
ip_vs_lblcr
ip_vs_sh
ip_vs_dh
ip_vs_sed
ip_vs_nq
nf_conntrack
EOF'

You can run the following command to load these modules on a machine that is already running.

sudo modprobe ip_vs
sudo modprobe ip_vs_rr
sudo modprobe ip_vs_wrr
sudo modprobe ip_vs_lc
sudo modprobe ip_vs_wlc
sudo modprobe ip_vs_lblc
sudo modprobe ip_vs_lblcr
sudo modprobe ip_vs_sh
sudo modprobe ip_vs_dh
sudo modprobe ip_vs_sed
sudo modprobe ip_vs_nq
sudo modprobe nf_conntrack

Note

It is highly recommended to execute these worker node steps as part of you worker node’s bootstrapping process via user data script or in any build scripts executed to build a custom worker node AMI.

Next, you will configure your cluster’s kube-proxy DaemonSet to run in IPVS mode. This is done by setting the kube-proxy mode to ipvs and the ipvs scheduler to one of the load balancing options listed above, for example: rr for Round Robin.

Warning

This is a disruptive change and should be performed in off-hours. We recommend making these changes during initial EKS cluster creation to minimize impacts.

You can issue an AWS CLI command to enable IPVS by updating the kube-proxy EKS Add-on.

aws eks update-addon --cluster-name $CLUSTER_NAME --addon-name kube-proxy \
  --configuration-values '{"ipvs": {"scheduler": "rr"}, "mode": "ipvs"}' \
  --resolve-conflicts OVERWRITE

Or you can do this by modifying the kube-proxy-config ConfigMap in your cluster.

kubectl -n kube-system edit cm kube-proxy-config

Find the scheduler setting under ipvs and set the value to one of the ipvs load balancing options listed above, for example: rr for Round Robin. Find the mode setting, which defaults to iptables, and change the value to ipvs. The result of either option should look similar to the configuration below.

  iptables:
    masqueradeAll: false
    masqueradeBit: 14
    minSyncPeriod: 0s
    syncPeriod: 30s
  ipvs:
    excludeCIDRs: null
    minSyncPeriod: 0s
    scheduler: "rr"
    syncPeriod: 30s
  kind: KubeProxyConfiguration
  metricsBindAddress: 0.0.0.0:10249
  mode: "ipvs"
  nodePortAddresses: null
  oomScoreAdj: -998
  portRange: ""
  udpIdleTimeout: 250ms

If your worker nodes were joined to your cluster prior to making these changes, you will need to restart the kube-proxy DaemonSet.

kubectl -n kube-system rollout restart ds kube-proxy

Validation

You can validate that your cluster and worker nodes are running in IPVS mode by issuing the following command on one of your worker nodes.

sudo ipvsadm -L

At a minimum, you should see a result similar to the one below, showing entries for the Kubernetes API Server service at 10.100.0.1 and the CoreDNS service at 10.100.0.10.

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  ip-10-100-0-1.us-east-1. rr
  -> ip-192-168-113-81.us-eas Masq        1      0          0
  -> ip-192-168-162-166.us-ea Masq        1      1          0
TCP  ip-10-100-0-10.us-east-1 rr
  -> ip-192-168-104-215.us-ea Masq        1      0          0
  -> ip-192-168-123-227.us-ea Masq        1      0          0
UDP  ip-10-100-0-10.us-east-1 rr
  -> ip-192-168-104-215.us-ea Masq        1      0          0
  -> ip-192-168-123-227.us-ea Masq        1      0          0

Note

This example output comes from an EKS cluster with a service IP address range of 10.100.0.0/16.