CNI custom networking
By default, when new network interfaces are allocated for pods, ipamD
-
There are a limited number of IP addresses available in a subnet. This might limit the number of pods that can be created in the cluster. Using different subnets for pods allows you to increase the number of available IP addresses.
-
For security reasons, your pods must use different security groups or subnets than the node's primary network interface.
-
The nodes are configured in public subnets and you want the pods to be placed in private subnets using a NAT Gateway. For more information, see External source network address translation (SNAT).
Considerations
-
The procedures in this topic require the Amazon VPC CNI plugin for Kubernetes
version 1.6.3-eksbuild.2 or later. -
Enabling a custom network effectively removes an available network interface (and all of its available IP addresses for pods) from each node that uses it. The primary network interface for the node is not used for pod placement when a custom network is enabled.
-
The procedure in this topic instructs the Amazon VPC CNI plugin to associate different security groups to secondary network interfaces than are associated to the primary network interface in the instance. All pods using the secondary network interfaces still share use of the secondary network interfaces and all use the same security groups.
If you want to assign different security groups to individual pods, then you can use Security groups for pods. Security groups for pods create additional network interfaces that can each be assigned a unique security group. Security groups for pods can be used with or without custom networking.
-
You can't use custom networking if you created your cluster to use the IPv6 family. If you plan to use custom networking to help alleviate IP address exhaustion, you can use IPv6 instead. For more information, see Assigning IPv6 addresses to pods and services.
To configure CNI custom networking
-
Confirm that your currently-installed Amazon VPC CNI plugin version is 1.6.3-eksbuild.2 or later.
kubectl describe daemonset aws-node \ --namespace kube-system | grep Image | cut -d "/" -f 2
Output:
amazon-k8s-cni:
1.6.3-eksbuild.2
If your version is earlier than 1.6.3-eksbuild.2, then you must update it. For more information, see the updating sections of Managing the Amazon VPC CNI add-on.
-
Associate a secondary CIDR block to your cluster's VPC. For more information, see Associating a Secondary IPv4 CIDR Block with Your VPC in the Amazon VPC User Guide.
-
Create a subnet in your VPC for each Availability Zone, using your secondary CIDR block. Your custom subnets must be from a different VPC CIDR block than the subnet that your nodes were launched into. For more information, see Creating a subnet in your VPC in the Amazon VPC User Guide.
-
Set the
AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
environment variable totrue
in theaws-node
DaemonSet:kubectl set env daemonset aws-node \ -n kube-system AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true
-
Create an
ENIConfig
custom resource for each subnet that you want to schedule pods in.-
Create a unique file for each network interface configuration. Each file must include the following contents with a unique value for
name
. We highly recommend using a value forname
that matches the Availability Zone of the subnet because this makes deployment of multi-Availability Zone Auto Scaling groups simpler (see step 5c below).In this example, a file named
is created. Replace theus-west-2a.yaml
example values
with your own values. In this example, we follow best practices and set the value forname
to the Availability Zone that the subnet is in. If you don't have a specific security group that you want to attach for your pods, you can leave that value empty for now. Later, you specify the node security group in the ENIConfig.apiVersion: crd.k8s.amazonaws.com/v1alpha1 kind: ENIConfig metadata: name:
us-west-2a
spec: securityGroups: -sg-0dff111a1d11c1c11
subnet:subnet-011b111c1f11fdf11
Note -
Each subnet and security group combination requires its own custom resource. If you have multiple subnets in the same Availability Zone, use the following command to annotate the nodes in each subnet with the matching config name.
kubectl annotate node \
node-name
.region-code
.compute.internal \ k8s.amazonaws.com/eniConfig=subnet1ConfigName
-
If you don't specify a valid security group for the VPC, and you're using version 1.8.0 or later of the VPC CNI plugin, then the security groups associated with the node's primary elastic network interface are used. If you're using a version of the plugin that is earlier than 1.8.0, then the default security group for the VPC is assigned to secondary elastic network interfaces.
-
If you specified a security group, ensure that the recommended or minimum required security group settings for the cluster, control plane and node security groups are met. For more information, see Amazon EKS security group considerations.
-
-
Apply each custom resource file that you created to your cluster with the following command:
kubectl apply -f
us-west-2a
.yaml -
(Optional, but recommended for multi-Availability Zone node groups) By default, Kubernetes applies the Availability Zone of a node to the
topology.kubernetes.io/zone
label. If you named yourENIConfig
custom resources after each Availability Zone in your VPC, as recommended in step 5a, then you can enable Kubernetes to automatically apply the correspondingENIConfig
for the node's Availability Zone with the following command.kubectl set env daemonset aws-node \ -n kube-system ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone
Note Ensure that an annotation with the key
k8s.amazonaws.com/eniConfig
for theENI_CONFIG_ANNOTATION_DEF
environment variable doesn't exist in the container spec for theaws-node
DaemonSet
. If it exists, it overrides theENI_CONFIG_LABEL_DEF
value, and should be removed. You can check to see if the variable is set with thekubectl describe daemonset aws-node -n kube-system | grep ENI_CONFIG_ANNOTATION_DEF
command. If no output is returned, then the variable is not set.
-
-
If you plan to deploy a managed node group without a launch template, or with a launch template that you haven't specified an AMI ID in, then skip to step 7 and use the Managed, Without a launch template or with a launch template without an AMI ID specified option. Managed node groups automatically calculates the maximum pods value for you.
If you're deploying a self-managed node group or a managed node group with a launch template that you have specified an AMI ID in, then you must determine the Amazon EKS recommend number of maximum pods for your nodes. Follow the instructions in Amazon EKS recommended maximum pods for each Amazon EC2 instance type, adding
to step 3. Note the output for use in a later step.--cni-custom-networking-enabled
-
Create one of the following types of node groups. For additional instance selection criteria, see Choosing an Amazon EC2 instance type. For the options that include
20
, replace it with either the value from the previous step (recommended) or your own value.-
Self-managed – Deploy the node group using the instructions in Launching self-managed Amazon Linux nodes. Don't specify the subnets that you specified in the
ENIConfig
resources that you deployed. Specify the following text for the BootstrapArguments parameter.--use-max-pods false --kubelet-extra-args '--max-pods=
'20
-
Managed – Deploy your node group using one of the following options:
-
Without a launch template or with a launch template without an AMI ID specified – Complete the procedure in Creating a managed node group. Managed node groups automatically calculates the Amazon EKS recommended
max-pods
value for you. -
With a launch template with a specified AMI ID – In your launch template, specify an Amazon EKS optimized AMI ID, or a custom AMI built off the Amazon EKS optimized AMI, then deploy the node group using a launch template and provide the following user data in the launch template. This user data passes arguments into the
bootstrap.sh
file. For more information about the bootstrap file, see bootstrap.shon GitHub. /etc/eks/bootstrap.sh
my-cluster
\ --use-max-pods false \ --kubelet-extra-args '--max-pods=20
'If you've created a custom AMI that is not built off the Amazon EKS optimized AMI, then you need to custom create the configuration yourself.
-
Note If you want your nodes to support a significantly higher number of pods, run the script in Amazon EKS recommended maximum pods for each Amazon EC2 instance type again, adding the
option to the command. For example,--cni-prefix-delegation-enabled
110
is returned for anm5.large
instance type. To enable this capability, see Increase the amount of available IP addresses for your Amazon EC2 nodes. You can use thiscapability with custom networking.
-
-
After your node group is created, record the security group that was created for the subnet and apply the security group to the associated
ENIConfig
. Edit eachENIConfig
with the following command, replacingeniconfig-name
with your value:kubectl edit eniconfig.crd.k8s.amazonaws.com/
eniconfig-name
If you followed best practices in step 5, the
eniconfig-name
corresponds to the Availability Zone name.The
spec
section should look like the following example spec:spec: securityGroups: -
sg-0dff222a2d22c2c22
subnet:subnet-022b222c2f22fdf22
Note If you use the security group that was created, ensure that the recommended or minimum required security group settings for the cluster, control plane and node security groups are met. For more information, see Amazon EKS security group considerations.
-
If you had any nodes in your cluster with running pods before you switched to the custom CNI networking feature, you should cordon and drain the nodes to gracefully shutdown the pods and then terminate the nodes. Only new nodes that are registered with the
k8s.amazonaws.com/eniConfig
label use the custom networking feature.