Customizing managed nodes with launch templates
For the highest level of customization, you can deploy managed nodes using your own launch template. Using a launch template allows capabilities such as the following:
-
Provide bootstrap arguments at deployment of a node, such as extra
kubelet
arguments. -
Assign IP addresses to Pods from a different CIDR block than the IP address assigned to the node.
-
Deploy your own custom AMI to nodes.
-
Deploy your own custom CNI to nodes.
When you give your own launch template upon first creating a managed node group, you will also have greater flexibility later. As long as you deploy a managed node group with your own launch template, you can iteratively update it with a different version of the same launch template. When you update your node group to a different version of your launch template, all nodes in the group are recycled to match the new configuration of the specified launch template version.
Managed node groups are always deployed with a launch template to be used with the Amazon EC2 Auto Scaling group. When you don't provide a launch template, the Amazon EKS API creates one automatically with default values in your account. However, we don't recommend that you modify auto-generated launch templates. Furthermore, existing node groups that don't use a custom launch template can't be updated directly. Instead, you must create a new node group with a custom launch template to do so.
Launch template configuration basics
You can create an Amazon EC2 Auto Scaling launch template with the AWS Management Console, AWS CLI, or an AWS SDK. For more information, see Creating a Launch Template for an Auto Scaling group in the Amazon EC2 Auto Scaling User Guide. Some of the settings in a launch template are similar to the settings used for managed node configuration. When deploying or updating a node group with a launch template, some settings must be specified in either the node group configuration or the launch template. Don't specify a setting in both places. If a setting exists where it shouldn't, then operations such as creating or updating a node group fail.
The following table lists the settings that are prohibited in a launch template. It also lists similar settings, if any are available, that are required in the managed node group configuration. The listed settings are the settings that appear in the console. They might have similar but different names in the AWS CLI and SDK.
Launch template – Prohibited | Amazon EKS node group configuration |
---|---|
Subnet under Network interfaces (Add network interface) | Subnets under Node group network configuration on the Specify networking page |
IAM instance profile under Advanced details | Node IAM role under Node group configuration on the Configure Node group page |
Shutdown behavior and Stop - Hibernate behavior under Advanced details. Retain default Don't include in launch template setting in launch template for both settings. | No equivalent. Amazon EKS must control the instance lifecycle, not the Auto Scaling group. |
The following table lists the prohibited settings in a managed node group configuration. It also lists similar settings, if any are available, which are required in a launch template. The listed settings are the settings that appear in the console. They might have similar names in the AWS CLI and SDK.
Amazon EKS node group configuration – Prohibited | Launch template |
---|---|
(Only if you specified a custom AMI in a launch template) AMI type under Node group compute configuration on Set compute and scaling configuration page – Console displays Specified in launch template and the AMI ID that was specified. If Application and OS Images (Amazon Machine Image) wasn't specified in the launch template, you can select an AMI in the node group configuration. |
Application and OS Images (Amazon Machine Image) under Launch template contents – You must specify an ID if you have either of the following requirements:
|
Disk size under Node group compute configuration on Set compute and scaling configuration page – Console displays Specified in launch template. | Size under Storage (Volumes) (Add new volume). You must specify this in the launch template. |
SSH key pair under Node group configuration on the Specify Networking page – The console displays the key that was specified in the launch template or displays Not specified in launch template. | Key pair name under Key pair (login). |
You can't specify source security groups that are allowed remote access when using a launch template. | Security groups under Network settings for the instance or Security groups under Network interfaces (Add network interface), but not both. For more information, see Using custom security groups. |
Note
-
If you deploy a node group using a launch template, specify zero or one Instance type under Launch template contents in a launch template. Alternatively, you can specify 0–20 instance types for Instance types on the Set compute and scaling configuration page in the console. Or, you can do so using other tools that use the Amazon EKS API. If you specify an instance type in a launch template, and use that launch template to deploy your node group, then you can't specify any instance types in the console or using other tools that use the Amazon EKS API. If you don't specify an instance type in a launch template, in the console, or using other tools that use the Amazon EKS API, the
t3.medium
instance type is used. If your node group is using the Spot capacity type, then we recommend specifying multiple instance types using the console. For more information, see Managed node group capacity types. -
If any containers that you deploy to the node group use the Instance Metadata Service Version 2, make sure to set the Metadata response hop limit to
2
in your launch template. For more information, see Instance metadata and user data in the Amazon EC2 User Guide for Linux Instances. If you deploy a managed node group without using a custom launch template, this value is automatically set for the node group in the default launch template.
Tagging Amazon EC2 instances
You can use the TagSpecification
parameter of a launch template to
specify which tags to apply to Amazon EC2 instances in your node group. The IAM entity
calling the CreateNodegroup
or UpdateNodegroupVersion
APIs
must have permissions for ec2:RunInstances
and
ec2:CreateTags
, and the tags must be added to the launch
template.
Using custom security groups
You can use a launch template to specify custom Amazon EC2 security groups to apply to instances in your node group. This can be either in the instance level security groups parameter or as part of the network interface configuration parameters. However, you can't create a launch template that specifies both instance level and network interface security groups. Consider the following conditions that apply to using custom security groups with managed node groups:
-
Amazon EKS only allows launch templates with a single network interface specification.
-
By default, Amazon EKS applies the cluster security group to the instances in your node group to facilitate communication between nodes and the control plane. If you specify custom security groups in the launch template using either option mentioned earlier, Amazon EKS doesn't add the cluster security group. So, you must ensure that the inbound and outbound rules of your security groups enable communication with the endpoint of your cluster. If your security group rules are incorrect, the worker nodes can't join the cluster. For more information about security group rules, see Amazon EKS security group requirements and considerations.
-
If you need SSH access to the instances in your node group, include a security group that allows that access.
Amazon EC2 user data
The launch template includes a section for custom user data. You can specify
configuration settings for your node group in this section without manually creating
individual custom AMIs. For more information about the settings available for
Bottlerocket, see Using user
data
You can supply Amazon EC2 user data in your launch template using
cloud-init
when launching your instances. For more information, see
the cloud-init
Amazon EC2 user data in launch templates that are used with managed node groups must be
in the MIME multi-part archivekubelet
. This is performed as part of the user
data merged by Amazon EKS. Certain kubelet
parameters, such as setting
labels on nodes, can be configured directly through the managed node groups
API.
Note
For more information about advanced kubelet
customization,
including manually starting it or passing in custom configuration parameters,
see Specifying an AMI. If a custom AMI ID is
specified in a launch template, Amazon EKS doesn't merge user data.
The following details provide more information about the user data section.
Specifying an AMI
If you have either of the following requirements, then specify an AMI ID in the
imageId
field of your launch template. Select the requirement you have
for additional information.
Bootstrapping is a term used to describe adding commands that can be run when
an instance starts. For example, bootstrapping allows using extra kubelet
bootstrap.sh
script by using eksctl
without specifying a
launch template. Or you can do so by specifying the information in the user data
section of a launch template.
Bootstrapping is a term used to describe adding commands that can be run when
an instance starts. You can pass arguments to the
Start-EKSBootstrap.ps1
script by using eksctl
without
specifying a launch template. Or you can do so by specifying the information in
the user data section of a launch template.
If you want to specify a custom Windows AMI ID, keep in mind the following considerations:
-
You must use a launch template and give the required bootstrap commands in the user data section. To retrieve your desired Windows ID, you can use the table in Amazon EKS optimized Windows AMIs.
-
There are several limits and conditions. For example, you must add
eks:kube-proxy-windows
to your AWS IAM Authenticator configuration map. For more information, see Limits and conditions when specifying an AMI ID.
Specify the following information in the user data section of your launch
template. Replace every
with your own values. The example value
-APIServerEndpoint
,
-Base64ClusterCA
, and -DNSClusterIP
arguments are
optional. However, defining them allows the Start-EKSBootstrap.ps1
script to avoid making a describeCluster
call.
-
The only required argument is the cluster name (
).my-cluster
-
To retrieve the
for your cluster, run the following command.certificate-authority
aws eks describe-cluster --query "cluster.certificateAuthority.data" --output text --name
my-cluster
--regionregion-code
-
To retrieve the
for your cluster, run the following command.api-server-endpoint
aws eks describe-cluster --query "cluster.endpoint" --output text --name
my-cluster
--regionregion-code
-
The value for
--dns-cluster-ip
is your service CIDR with.10
at the end. To retrieve the
for your cluster, run the following command. For example, if the returned value for isservice-cidr
ipv4 10.100.0.0/16
, then your value is
.10.100.0.10
aws eks describe-cluster --query "cluster.kubernetesNetworkConfig.serviceIpv4Cidr" --output text --name
my-cluster
--regionregion-code
-
This example creates a node group using
as the runtime, but you can modify it as needed.containerd
For additional arguments, see Bootstrap script configuration parameters.
Note
If you're using custom service CIDR, then you need to specify it using the
-ServiceCIDR
parameter. Otherwise, the DNS resolution for Pods in the cluster will fail.
<powershell> [string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1" & $EKSBootstrapScriptFile -EKSClusterName
my-cluster
` -Base64ClusterCAcertificate-authority
` -APIServerEndpointapi-server-endpoint
` -DNSClusterIPservice-cidr.10
` -ContainerRuntimecontainerd
</powershell>
For more information, see Amazon Machine Images (AMI)
in the Amazon EC2 User Guide for Linux Instances. The Amazon EKS AMI build
specification contains resources and configuration scripts for building a custom
Amazon EKS AMI based on Amazon Linux 2. For more information, see Amazon EKS AMI Build
Specification
Important
When specifying an AMI, Amazon EKS doesn't merge any user data. Rather, you're
responsible for supplying the required bootstrap
commands for nodes to
join the cluster. If your nodes fail to join the cluster, the Amazon EKS
CreateNodegroup
and UpdateNodegroupVersion
actions
also fail.
Limits and conditions when specifying an AMI ID
The following are the limits and conditions involved with specifying an AMI ID with managed node groups:
-
You must create a new node group to switch between specifying an AMI ID in a launch template and not specifying an AMI ID.
-
You aren't notified in the console when a newer AMI version is available. To update your node group to a newer AMI version, you need to create a new version of your launch template with an updated AMI ID. Then, you need to update the node group with the new launch template version.
-
The following fields can't be set in the API if you specify an AMI ID:
-
amiType
-
releaseVersion
-
version
-
-
Any
taints
set in the API are applied asynchronously if you specify an AMI ID. To apply taints prior to a node joining the cluster, you must pass the taints tokubelet
in your user data using the--register-with-taints
command line flag. For more information, seekubelet
in the Kubernetes documentation. -
When specifying a custom AMI ID for Windows managed node groups, add
eks:kube-proxy-windows
to your AWS IAM Authenticator configuration map. This is required for DNS to function properly.-
Open the AWS IAM Authenticator configuration map for editing.
kubectl edit -n kube-system cm aws-auth
-
Add this entry to the
groups
list under eachrolearn
associated with Windows nodes. Your configuration map should look similar toaws-auth-cm-windows.yaml
. - eks:kube-proxy-windows
-
Save the file and exit your text editor.
-