Increase the amount of available IP addresses for your Amazon EC2 nodes
Each Amazon EC2 instance supports a maximum number of elastic network interfaces and a maximum
number of IP addresses that can be assigned to each network interface. Each node requires one IP
address for each network interface. All other available IP addresses can be assigned to
Pods
. Each Pod
requires its own IP address. As a result, you might have
nodes that have available compute and memory resources, but can't accommodate additional
Pods
because the node has run out of IP addresses to assign to
Pods
.
In this topic, you learn how to significantly increase the number of IP addresses that nodes
can assign to Pods
by assigning IP prefixes, rather than assigning individual
secondary IP addresses to your nodes. Each prefix includes several IP addresses. If you don't
configure your cluster for IP prefix assignment, your cluster must make more Amazon EC2 application
programming interface (API) calls to configure network interfaces and IP addresses necessary for
Pod connectivity. As clusters grow to larger sizes, the frequency of these API
calls can lead to longer Pod and instance launch times. This results in scaling
delays to meet the demand of large and spiky workloads, and adds cost and management overhead
because you need to provision additional clusters and VPCs to meet scaling requirements. For more
information, see Kubernetes Scalability thresholds
Considerations
-
Each Amazon EC2 instance type supports a maximum number of Pods. If your managed node group consists of multiple instance types, the smallest number of maximum Pods for an instance in the cluster is applied to all nodes in the cluster.
-
By default, the maximum number of
Pods
that you can run on a node is 110, but you can change that number. If you change the number and have an existing managed node group, the next AMI or launch template update of your node group results in new nodes coming up with the changed value. -
When transitioning from assigning IP addresses to assigning IP prefixes, we recommend that you create new node groups to increase the number of available IP addresses, rather than doing a rolling replacement of existing nodes. Running Pods on a node that has both IP addresses and prefixes assigned can lead to inconsistency in the advertised IP address capacity, impacting the future workloads on the node. For the recommended way of performing the transition, see Replace all nodes during migration from Secondary IP mode to Prefix Delegation mode or vice versa
in the Amazon EKS best practices guide. -
For clusters with Linux nodes only.
-
Once you configure the add-on to assign prefixes to network interfaces, you can't downgrade your Amazon VPC CNI plugin for Kubernetes add-on to a version lower than
1.9.0
(or1.10.1
) without removing all nodes in all node groups in your cluster. -
If you're also using security groups for Pods, with
POD_SECURITY_GROUP_ENFORCING_MODE
=standard
andAWS_VPC_K8S_CNI_EXTERNALSNAT
=false
, when your Pods communicate with endpoints outside of your VPC, the node's security groups are used, rather than any security groups you've assigned to your Pods.If you're also using security groups for Pods, with
POD_SECURITY_GROUP_ENFORCING_MODE
=strict
, when yourPods
communicate with endpoints outside of your VPC, thePod's
security groups are used.
-
Prerequisites
-
An existing cluster. To deploy one, see Creating an Amazon EKS cluster.
-
The subnets that your Amazon EKS nodes are in must have sufficient contiguous
/28
(forIPv4
clusters) or/80
(forIPv6
clusters) Classless Inter-Domain Routing (CIDR) blocks. You can only have Linux nodes in anIPv6
cluster. Using IP prefixes can fail if IP addresses are scattered throughout the subnet CIDR. We recommend that following:-
Using a subnet CIDR reservation so that even if any IP addresses within the reserved range are still in use, upon their release, the IP addresses aren't reassigned. This ensures that prefixes are available for allocation without segmentation.
-
Use new subnets that are specifically used for running the workloads that IP prefixes are assigned to. Both Windows and Linux workloads can run in the same subnet when assigning IP prefixes.
-
-
To assign IP prefixes to your nodes, your nodes must be AWS Nitro-based. Instances that aren't Nitro-based continue to allocate individual secondary IP addresses, but have a significantly lower number of IP addresses to assign to Pods than Nitro-based instances do.
-
For clusters with Linux nodes only – If your cluster is configured for the
IPv4
family, you must have version1.9.0
or later of the Amazon VPC CNI plugin for Kubernetes add-on installed. You can check your current version with the following command.kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2
If your cluster is configured for the
IPv6
family, you must have version1.10.1
of the add-on installed. If your plugin version is earlier than the required versions, you must update it. For more information, see the updating sections of Working with the Amazon VPC CNI plugin for Kubernetes Amazon EKS add-on. -
For clusters with Windows nodes only
-
Your cluster and its platform version must be at, or later than the versions in the following table. To upgrade your cluster version, see Updating an Amazon EKS cluster Kubernetes version. If your cluster isn't at the minimum platform version, then you can't assign IP prefixes to your nodes until Amazon EKS has updated your platform version.
Kubernetes version Platform version 1.27
eks.3
1.26
eks.4
1.25
eks.5
You can check your current Kubernetes and platform version by replacing
in the following command with the name of your cluster and then running the modified command:my-cluster
aws eks describe-cluster --name
.my-cluster
--query 'cluster.{"Kubernetes Version": version, "Platform Version": platformVersion}' -
Windows support enabled for your cluster. For more information, see Enabling Windows support for your Amazon EKS cluster.
-
To increase the amount of available IP addresses for your Amazon EC2 nodes
-
Configure your cluster to assign IP address prefixes to nodes. Complete the procedure on the tab that matches your node's operating system.
-
Once your nodes are deployed, view the nodes in your cluster.
kubectl get nodes
An example output is as follows.
NAME STATUS ROLES AGE VERSION ip-
192-168-22-103
.region-code
.compute.internal Ready <none>19m
v1.XX.X-eks-6b7464
ip-192-168-97-94
.region-code
.compute.internal Ready <none>19m
v1.XX.X-eks-6b7464
-
Describe one of the nodes to determine the value of
max-pods
for the node and the number of available IP addresses. Replace
with the192.168.30.193
IPv4
address in the name of one of your nodes returned in the previous output.kubectl describe node ip-
192-168-30-193
.region-code
.compute.internal | grep 'pods\|PrivateIPv4Address'An example output is as follows.
pods:
110
vpc.amazonaws.com/PrivateIPv4Address:144
In the previous output,
110
is the maximum number of Pods that Kubernetes will deploy to the node, even though144
IP addresses are available.