Help improve this page
To contribute to this user guide, choose the Edit this page on GitHub link that is located in the right pane of every page.
Target AI/ML workloads to specific nodes with labels
Overview
An AI/ML cluster is rarely homogeneous. A single cluster often mixes general-purpose nodes that run platform components, GPU nodes of more than one model, and AWS Trainium or AWS Inferentia nodes, provisioned through more than one path. Getting a Pod onto the right subset of that fleet is a scheduling problem, and the inputs the scheduler uses are node labels.
Amazon EKS and the components that run on your nodes apply a large set of labels automatically. You do not have to create them, and most customers never need to add their own. This recipe covers which labels exist, which component applies each one, and how to match them from a workload.
Labels are not uniform across compute types. EKS managed node groups, Karpenter, and EKS Auto Mode express several of the same concepts with different label keys, and in one case with differently formatted values. If you are moving a workload between compute types or writing a manifest for a mixed cluster, start with Same concept, different label by provisioner.
Important
Labels constrain which node a Pod lands on. They do not reserve or allocate a device. To use an accelerator, the Pod must also request the device as a resource (for example, nvidia.com/gpu or aws.amazon.com/neuron), or claim it through Dynamic Resource Allocation. A Pod that only matches eks.amazonaws.com/instance-gpu-name lands on a GPU node without being granted a GPU. For more information, see Manage hardware devices on Amazon EKS.
Labels available for targeting
Which labels a node carries depends on how the node was provisioned and which components run on it. A label is present only when the component that writes it runs on the node.
Labels on every node
These labels are applied by kubelet and by the AWS cloud controller manager, which Amazon EKS runs in the cluster control plane. They are present regardless of how you provisioned the node.
| Label | Example | Description |
|---|---|---|
|
|
|
The AWS Region. |
|
|
|
The Availability Zone name. Zone names are mapped for each account, so the same name can refer to different physical zones in different accounts. |
|
|
|
The Availability Zone ID. Unlike the zone name, the ID identifies the same physical zone in every account. Use it when you need to correlate placement across accounts. |
|
|
|
The EC2 instance type. |
|
|
|
CPU architecture and operating system. |
|
|
|
The Kubernetes node name. See The node name format differs by provisioner. |
Labels on accelerated and HPC nodes only
The AWS cloud controller manager also applies the topology.k8s.aws/network-node-layer-* labels, which describe where the node sits in the AWS network hierarchy. These labels are not on every node. They are present only on the accelerated, HPC, and GPU instance types for which Amazon EC2 reports topology, and only in the AWS Regions where that API is available.
| Label | Example | Description |
|---|---|---|
|
|
|
The node’s position in the AWS network hierarchy, from the top layer down. Present on the |
|
|
|
A fourth, more specific layer. Present only on |
A node of any other instance type, such as m5.large, carries no topology.k8s.aws/network-node-layer-* labels at all. Do not write a selector that assumes these labels exist across a mixed cluster. For the authoritative per-size list of supported instance types, and for how to use these labels to place the Pods of a distributed job close to each other, see Place distributed AI/ML workloads with network topology labels.
Labels that identify how a node was provisioned
Use these labels to separate workloads across compute types in a mixed cluster, or to pin a workload to one node group or NodePool. Choose the tab for the way you provision nodes.
Same concept, different label by provisioner
Several concepts are expressed by a different label, or by the same label with differently formatted values, depending on how the node was provisioned. A manifest that works on one compute type can leave Pods Pending on another. Use the following table when you move a workload between compute types, or when you write a manifest that has to work across a mixed cluster.
| Concept | EKS Auto Mode | Karpenter | EKS managed node groups |
|---|---|---|---|
|
Which group or pool owns the node |
|
|
|
|
Which template configured the node |
|
|
|
|
Capacity type |
|
|
|
|
Capacity reservation ID |
|
|
No label. Match the network topology instead, as described in Place distributed AI/ML workloads with network topology labels. |
|
AMI the node booted from |
No label. |
No label. |
|
|
Instance attributes, such as CPU count, memory, and GPU model |
|
|
No labels. Match |
|
Instance tenancy |
|
|
No label. |
|
Node name, in |
The EC2 instance ID, such as |
Varies with the AMI family and node configuration. Read the label from the cluster. |
The EC2 private DNS name, such as |
|
Position in the AWS network |
|
|
|
Two implications are worth calling out:
-
The capacity type values are not interchangeable.
eks.amazonaws.com/capacityType: ON_DEMANDandkarpenter.sh/capacity-type: on-demanddescribe the same thing in different formats. A selector written for one silently matches nothing on the other, because a label mismatch is not an error. -
Self-managed nodes carry only what you set. The
eks.amazonaws.com/*labels in the managed node group column are applied by Amazon EKS through the node group’s launch template user data. A self-managed node gets them only if you pass them to kubelet yourself, so treat the managed node group column as a starting point rather than a guarantee.
EKS Auto Mode nodes also carry app.kubernetes.io/managed-by: eks and topology.ebs.csi.eks.amazonaws.com/zone. These identify the managing service and the zone used by the EBS CSI driver. They are not intended as scheduling constraints.
Labels that describe instance attributes
EKS Auto Mode and Karpenter compute a set of labels from the EC2 instance type, covering CPU count, memory, network and EBS bandwidth, local NVMe, GPU model and memory, and accelerator name and count. Managed node groups and self-managed nodes do not apply these labels; match node.kubernetes.io/instance-type instead, or add your own labels to the node group.
The two provisioners use different label prefixes for the same attributes. EKS Auto Mode uses eks.amazonaws.com/instance-
and Karpenter uses karpenter.k8s.aws/instance-.
-
For the subset of these labels that matters most for AI/ML targeting, shown side by side for both prefixes, see Common AI/ML well-known labels.
-
For the full list of EKS Auto Mode labels, see EKS Auto Mode Supported Labels.
-
For the full list of Karpenter labels, see Well-Known Labels
in the Karpenter documentation.
Labels that describe devices
Device plugins and feature-discovery components apply their own labels after the node joins.
-
NVIDIA GPU nodes carry
nvidia.com/*labels that describe the GPU model, count, memory, MIG configuration, and sharing strategy. For which component applies each label and how to interpret them, see NVIDIA GPU node labels. -
EKS Auto Mode applies
neuron.amazonaws.com/presentto nodes with AWS Trainium or AWS Inferentia accelerators.
The node name format differs by provisioner
The value of kubernetes.io/hostname is the Kubernetes node name, and the node name format is not the same across provisioners:
-
Managed node groups and self-managed nodes use the EC2 private DNS name, such as
ip-192-168-3-238.us-west-2.compute.internal. -
EKS Auto Mode uses the EC2 instance ID, such as
i-0c4bf4493cf51d74c.
A manifest that hardcodes one format leaves Pods Pending when you move it to the other compute type. Read the label from the cluster rather than deriving it from an instance ID or an IP address.
The node name is not the only label that differs between compute types. For the others, see Same concept, different label by provisioner.
Discover the labels on your own nodes
The authoritative list for your cluster is the cluster itself. List every label on a node:
kubectl get node <node-name> -o go-template='{{range $k, $v := .metadata.labels}}{{$k}} = {{$v}}{{"\n"}}{{end}}' | sort
Show selected labels as columns across all nodes:
kubectl get nodes \ -L node.kubernetes.io/instance-type \ -L karpenter.sh/capacity-type \ -L eks.amazonaws.com/nodegroup \ -L topology.k8s.aws/zone-id
List the nodes that match a prospective selector before you deploy against it:
kubectl get nodes -l eks.amazonaws.com/instance-gpu-manufacturer=nvidia
Considerations
-
Prefer capability labels to name labels. Matching
eks.amazonaws.com/instance-gpu-name: rtx-pro-server-6000describes what the workload needs. Matchingkarpenter.sh/nodepool: gpu-pooloreks.amazonaws.com/nodegroup: gpu-ngties the workload to a specific piece of infrastructure, and leaves PodsPendingif you rename or replace it. For capacity that you provision yourself, define your own label on the node group or NodePool, such asworkload-class: gpu-inference, and match that. -
Reserve
eks.amazonaws.com,karpenter.sh,karpenter.k8s.aws, andtopology.k8s.awsfor Amazon EKS. These prefixes are managed. Do not set labels in them yourself; a label you add can be overwritten or can conflict with provisioning logic. Use your own prefix for custom labels. -
Labels are not a substitute for taints. A label keeps a workload off the wrong nodes only if every workload opts in. To keep workloads that do not request a GPU away from expensive GPU nodes by default, taint the nodes. For more information, see Prevent Pods from being scheduled on accelerated nodes.
-
Node labels are mutable, and some are reconciled. Labels you add with
kubectl labelpersist, but a label that a provisioner manages can be reset when the node is reconciled or replaced. Set durable custom labels in the node group or NodePool definition instead, so every replacement node inherits them. -
Deprecated aliases exist. Nodes also carry
beta.kubernetes.io/arch,beta.kubernetes.io/instance-type,beta.kubernetes.io/os, andfailure-domain.beta.kubernetes.io/*. These are deprecated aliases. Use the non-beta labels in new manifests.
Prerequisites
-
An existing Amazon EKS cluster with nodes.
-
kubectlconfigured to communicate with your cluster. For more information, see Install or update kubectl.
Step 1: Target a capability with nodeSelector
nodeSelector is the simplest form of targeting. Every key-value pair must match, and a Pod with no matching node stays Pending. Use it when the constraint is absolute and expressible as equality.
The following Pod runs only on an EKS Auto Mode node with an NVIDIA GPU.
apiVersion: v1 kind: Pod metadata: name: inference spec: nodeSelector: eks.amazonaws.com/compute-type: auto eks.amazonaws.com/instance-gpu-manufacturer: nvidia tolerations: - key: nvidia.com/gpu operator: Exists effect: NoSchedule containers: - name: app image: public.ecr.aws/docker/library/busybox:latest command: ["sh", "-c", "sleep 3600"] resources: limits: nvidia.com/gpu: "1"
Three separate things are doing work here, and a GPU workload needs all three:
-
The
nodeSelectordirects the Pod to a GPU node. -
The
tolerationallows it onto a node that is tainted to keep non-GPU workloads off. If your GPU nodes are not tainted, the toleration is harmless. For the taint side, see Prevent Pods from being scheduled on accelerated nodes. -
The
nvidia.com/gpulimit allocates a GPU on the node. Without it the Pod lands on a GPU node without being granted a GPU.
For AWS Trainium and AWS Inferentia nodes, the same pattern applies with the eks.amazonaws.com/instance-accelerator-manufacturer label and the aws.amazon.com/neuron resource. For more information, see Manage Neuron devices on Amazon EKS.
Step 2: Express hard and soft constraints with node affinity
Node affinity is more expressive than nodeSelector. It supports set operators such as In, NotIn, Exists, and Gt, and it distinguishes required constraints from preferences.
-
requiredDuringSchedulingIgnoredDuringExecutionis a hard constraint. A Pod with no matching node staysPending. -
preferredDuringSchedulingIgnoredDuringExecutionis a soft constraint. The scheduler weights matching nodes higher but places the Pod elsewhere rather than leaving itPending.
Combine both to express "an NVIDIA GPU is required, and one particular GPU model is preferred":
apiVersion: apps/v1 kind: Deployment metadata: name: inference spec: replicas: 4 selector: matchLabels: app: inference template: metadata: labels: app: inference spec: affinity: nodeAffinity: # Hard: the node must have an NVIDIA GPU. requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: eks.amazonaws.com/instance-gpu-manufacturer operator: In values: ["nvidia"] # Soft: prefer RTX PRO 6000 Blackwell GPUs, fall back to any other GPU node. preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 preference: matchExpressions: - key: eks.amazonaws.com/instance-gpu-name operator: In values: ["rtx-pro-server-6000"] containers: - name: app image: public.ecr.aws/docker/library/busybox:latest command: ["sh", "-c", "sleep 3600"] resources: limits: nvidia.com/gpu: "1"
Note
Within a single matchExpressions list, all expressions must match, which is a logical AND. Separate entries in nodeSelectorTerms are ORed. To express "this GPU model or that one", put both values in one expression’s values list rather than writing two terms.
Step 3: Pin a Pod to one host
Pin to kubernetes.io/hostname when you need a specific machine, for example to reproduce a failure on one node or to colocate a debugging Pod with a running job. Use the node name format for your provisioner, as described in The node name format differs by provisioner.
apiVersion: v1 kind: Pod metadata: name: debug spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: ["ip-192-168-3-238.us-west-2.compute.internal"] containers: - name: app image: public.ecr.aws/docker/library/busybox:latest command: ["sh", "-c", "sleep 3600"]
Host pinning is appropriate for short-lived, operational Pods. Avoid it for production workloads: the Pod cannot be rescheduled if that node is replaced during a node group update, a Karpenter consolidation, or node auto repair.
Step 4: Spread replicas across hosts
To keep replicas from concentrating on one node, use a topology spread constraint rather than affinity. The following constraint allows at most one replica of difference between any two nodes, and leaves excess replicas Pending rather than stacking them.
topologySpreadConstraints: - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: inference
Set whenUnsatisfiable: ScheduleAnyway to make the spread a preference instead.
Any label can serve as a topologyKey. Be clear about which direction you want before choosing one: a spread constraint distributes replicas across domains, so pointing it at a topology.k8s.aws/network-node-layer-* key scatters a job rather than co-locating it. To pull the Pods of a distributed job into one network domain, use Pod affinity instead, as described in Place distributed AI/ML workloads with network topology labels.
What happens when no node matches
Node affinity and nodeSelector are evaluated only when a Pod is scheduled. IgnoredDuringExecution in the affinity field names means exactly that: a running Pod is never re-evaluated, even if the node’s labels change or a better node appears.
The following table summarizes the behavior when the target nodes become unavailable and then return. The hard constraint is requiredDuringSchedulingIgnoredDuringExecution and the soft constraint is preferredDuringSchedulingIgnoredDuringExecution.
| Situation | Hard constraint | Soft constraint |
|---|---|---|
|
No matching node at scheduling time |
Pod stays |
Pod is scheduled onto any node that fits. |
|
A matching node appears later |
The scheduler binds the |
No effect. An already-running Pod is not moved onto the newly available preferred node. |
|
The node a running Pod is on is terminated |
The replacement Pod is scheduled against the constraint again, and stays |
The replacement Pod is scheduled against the preference again. |
If a Pod is stuck Pending, the scheduler’s reason is in the Pod’s events:
kubectl describe pod <pod-name>
Related topics
For more information, see the following topics.
-
Place distributed AI/ML workloads with network topology labels — Place the Pods of a distributed job close to each other on the network.
-
Prevent Pods from being scheduled on accelerated nodes — Keep workloads off specialized nodes with taints.
-
Control if a workload is deployed on EKS Auto Mode nodes — Require or exclude EKS Auto Mode nodes.
-
EKS Auto Mode Supported Labels — The full list of EKS Auto Mode instance-attribute labels.
-
NVIDIA GPU node labels — NVIDIA GPU node labels and the component that applies each one.
-
Assigning Pods to Nodes
in the Kubernetes documentation.
