**Help improve this page** To contribute to this user guide, choose the **Edit this page on GitHub** link that is located in the right pane of every page. # Prevent Pods from being scheduled on accelerated nodes ## Overview Nodes with specialized processors, such as GPUs, can be more expensive to run than nodes on standard machines. To protect these nodes from workloads that don’t require special hardware, you can use Kubernetes taints. Taints mark nodes to repel Pods that don’t have matching tolerations, ensuring only compatible workloads are scheduled. For more information, see [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) in the Kubernetes documentation. A taint is the complement of a node label. A label lets a workload opt *in* to a kind of node, but it does nothing to keep other workloads off. Only a taint makes exclusion the default, so that a Pod has to declare a toleration to land on the node. Use both together: taint the accelerated nodes so nothing lands there by accident, and label them so the workloads that belong there can find them. For the label side, see [Target AI/ML workloads to specific nodes with labels](ml-workload-placement.md). ### A convention for the taint key Any key works, but a consistent one is worth more than a descriptive one, because every workload and every add-on DaemonSet that needs an accelerated node has to spell the same key. The convention used throughout the Amazon EKS AI/ML guidance is to taint on the resource the node provides, with no value: ``` taints: - key: nvidia.com/gpu effect: NoSchedule ``` Workloads then tolerate it with `operator: Exists`, which matches the key regardless of value: ``` tolerations: - key: nvidia.com/gpu operator: Exists effect: NoSchedule ``` This pairing is what the Terraform and manifest examples in the [sample-eks-docs](https://github.com/aws-samples/sample-eks-docs) repository on GitHub use for every GPU node pool and GPU workload. Reusing it means the manifests in that repository, and the add-on Helm values that ship with it, work against your cluster without modification. For AWS Trainium and AWS Inferentia nodes, apply the same pattern with the `aws.amazon.com/neuron` key. ## Considerations + Taints applied through a NodePool are reconciled. If you add or remove a taint directly on a node with `kubectl taint`, EKS Auto Mode or Karpenter restores the NodePool’s definition. Change the NodePool, not the node. Nodes provisioned by EKS Auto Mode and Karpenter carry the `karpenter.sh/do-not-sync-taints` label, which governs this behavior. + Changing a NodePool’s taints replaces nodes. The change is drift, so existing nodes are drained and replaced rather than updated in place. Managed node groups behave differently, and update the taint on running nodes. + Taints can be updated after you create a managed node group using the `UpdateNodegroupConfig` API. + The taint key must begin with a letter or number. It can contain letters, numbers, hyphens (`-`), periods (`.`), and underscores (`_`). It can be up to 63 characters long. + Optionally, the taint key can begin with a DNS subdomain prefix and a single `/`. If it begins with a DNS subdomain prefix, it can be 253 characters long. The `nvidia.com/gpu` key used in this recipe takes this form. + The value is optional and must begin with a letter or number. It can contain letters, numbers, hyphens (`-`), periods (`.`), and underscores (`_`). It can be up to 63 characters long. + When using Kubernetes directly or the AWS Management Console, the taint effect must be `NoSchedule`, `PreferNoSchedule`, or `NoExecute`. However, when using the AWS CLI or API, the taint effect must be `NO_SCHEDULE`, `PREFER_NO_SCHEDULE`, or `NO_EXECUTE`. + A maximum of 50 taints are allowed per node group. + If taints that were created using a managed node group are removed manually from a node, then Amazon EKS doesn’t add the taints back to the node. This is true even if the taints are specified in the managed node group configuration. + `NoExecute` evicts Pods that are already running and do not tolerate the taint, unlike `NoSchedule`, which only affects new scheduling decisions. Use `NoSchedule` unless you intend to evict running workloads. ## Prerequisites + An [existing Amazon EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html). + [AWS CLI installed and configured](https://docs.aws.amazon.com/eks/latest/userguide/setting-up.html) with appropriate permissions. + `kubectl` configured to communicate with your cluster. For more information, see [Install or update `kubectl`](install-kubectl.md#kubectl-install-update). ## Step 1: Taint the accelerated nodes Where you declare the taint depends on how you provision nodes. In all three cases the taint is applied as the node joins the cluster, so there is no window in which an untainted accelerated node is schedulable. Choose the tab for the way you provision nodes. ------ #### [ EKS Auto Mode ] Declare the taint under `spec.template.spec.taints` on the NodePool. Every node the NodePool provisions carries it. ``` apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: gpu spec: template: metadata: labels: workload-class: gpu-inference spec: nodeClassRef: group: eks.amazonaws.com kind: NodeClass name: default taints: - key: nvidia.com/gpu effect: NoSchedule requirements: - key: eks.amazonaws.com/instance-category operator: In values: ["g"] - key: eks.amazonaws.com/instance-generation operator: Gt values: ["5"] - key: karpenter.sh/capacity-type operator: In values: ["on-demand"] ``` Apply the change with `kubectl apply`. Existing nodes are not retroactively tainted; EKS Auto Mode treats the change as drift and replaces them. For more information about NodePool configuration, see [Create a Node Pool for EKS Auto Mode](create-node-pool.md). ------ #### [ Self-managed Karpenter ] The field is the same as for EKS Auto Mode, because both use the `karpenter.sh/v1` NodePool API. Only the `nodeClassRef` and the instance requirement label prefix differ. ``` apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: gpu spec: template: metadata: labels: workload-class: gpu-inference spec: nodeClassRef: group: karpenter.k8s.aws kind: EC2NodeClass name: default taints: - key: nvidia.com/gpu effect: NoSchedule requirements: - key: karpenter.k8s.aws/instance-category operator: In values: ["g"] - key: karpenter.k8s.aws/instance-generation operator: Gt values: ["5"] - key: karpenter.sh/capacity-type operator: In values: ["on-demand"] ``` ------ #### [ EKS managed node groups ] Kubernetes node taints can be applied to new and existing managed node groups using the AWS Management Console or through the Amazon EKS API. The following example uses the AWS CLI. For information about creating a node group with a taint using the AWS Management Console, see [Create a managed node group for your cluster](create-managed-node-group.md). Note the `NO_SCHEDULE` spelling. The Amazon EKS API uses uppercase, underscore-separated effects, unlike the Kubernetes `NoSchedule` form used in the NodePool examples. ``` aws eks create-nodegroup \ --cli-input-json ' { "clusterName": "my-cluster", "nodegroupName": "gpu", "subnets": [ "subnet-1234567890abcdef0", "subnet-abcdef01234567890", "subnet-021345abcdef67890" ], "nodeRole": "arn:aws:iam::111122223333:role/AmazonEKSNodeRole", "instanceTypes": ["g6e.2xlarge"], "amiType": "AL2023_x86_64_NVIDIA", "labels": { "workload-class": "gpu-inference" }, "taints": [ { "key": "nvidia.com/gpu", "effect": "NO_SCHEDULE" } ] }' ``` To add, remove, or replace taints on an existing node group, use the [aws eks update-nodegroup-config](https://docs.aws.amazon.com/cli/latest/reference/eks/update-nodegroup-config.html) command with `addOrUpdateTaints` or `removeTaints`. Unlike the NodePool paths, this updates the taint on nodes that are already running, without replacing them. ``` aws eks update-nodegroup-config \ --cluster-name my-cluster \ --nodegroup-name gpu \ --taints 'addOrUpdateTaints=[{key=nvidia.com/gpu,effect=NO_SCHEDULE}]' ``` ------ Confirm the taint reached the nodes: ``` kubectl get nodes -o custom-columns='NODE:.metadata.name,TAINTS:.spec.taints[*].key' ``` ## Step 2: Tolerate the taint in your workload A tainted node repels every Pod that does not tolerate it, including your own GPU workloads. Add the toleration to the Pod spec, and pair it with a node selector so the Pod is both *allowed* onto the accelerated nodes and *directed* to them. ``` apiVersion: apps/v1 kind: Deployment metadata: name: inference spec: replicas: 1 selector: matchLabels: app: inference template: metadata: labels: app: inference spec: tolerations: - key: nvidia.com/gpu operator: Exists effect: NoSchedule nodeSelector: workload-class: gpu-inference containers: - name: app image: public.ecr.aws/docker/library/busybox:latest command: ["sh", "-c", "sleep 3600"] resources: limits: nvidia.com/gpu: "1" ``` **Important** A toleration on its own does not place a Pod. It only removes the objection. Without the `nodeSelector`, this Pod is free to schedule anywhere in the cluster, including on general-purpose nodes where the `nvidia.com/gpu` limit cannot be satisfied and the Pod stays `Pending`. Taints and labels do different jobs, and a GPU workload needs both. ## Step 3: Give add-on DaemonSets a matching toleration Tainting accelerated nodes also repels the DaemonSets that have to run *on* those nodes, such as device plugins, EFA plugins, and metrics exporters. A device plugin that cannot schedule onto a GPU node never advertises the GPU resource, so the node reports no allocatable GPUs and your workloads stay `Pending`. This failure is easy to misread, because the node looks healthy. EKS Auto Mode manages its own device plugins and tolerates its NodePool taints without configuration. For any component you install yourself, add the toleration to the chart values or the DaemonSet spec: ``` tolerations: - key: nvidia.com/gpu operator: Exists effect: NoSchedule ``` If workloads are `Pending` on a node that looks healthy, check whether the node advertises the resource at all: ``` kubectl get node -o jsonpath='{.status.allocatable}' ``` ## Related topics Taints repel workloads from nodes. To steer workloads toward nodes, use node labels: + [Target AI/ML workloads to specific nodes with labels](ml-workload-placement.md) — Target workloads to the right nodes using the labels that Amazon EKS applies automatically. + [Place distributed AI/ML workloads with network topology labels](ml-workload-topology.md) — Place the Pods of a distributed job close to each other using network topology labels. + [kubectl taint](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#taint) in the Kubernetes reference documentation.