How the repair action is chosen Replacement sequence on EKS Auto Mode and Karpenter Repair behavior on Karpenter v1.15.0 and later Configure automatic node repair

Automatically repair nodes in EKS clusters

This topic details the EKS automatic node repair behavior and how to configure it to meet your requirements. EKS automatic node repair is enabled by default in EKS Auto Mode, and can be used with EKS managed node groups and Karpenter.

The default EKS automatic node repair actions are summarized in the table below. They apply to EKS Auto Mode, EKS managed node groups, and self-managed Karpenter, with two exceptions on self-managed Karpenter v1.15.0 and later that are described in Repair behavior on Karpenter v1.15.0 and later.

For a detailed list of node health issues detected by the EKS node monitoring agent, their corresponding repair actions, and the per-code actions behind the AcceleratedHardwareReady condition, see Detect node health issues with the EKS node monitoring agent.

Node Condition	Description	Repair after	Repair action(s)
AcceleratedHardwareReady	AcceleratedHardwareReady indicates whether accelerated hardware (GPU, Neuron) on the node is functioning correctly.	10m	Replace or Reboot, depending on the condition reason. `Reboot` requires EKS managed node groups, or self-managed Karpenter `v1.15.0` and later with the `RebootForRepair` feature gate enabled. See NVIDIA XID error codes.
ContainerRuntimeReady	ContainerRuntimeReady indicates whether the container runtime (containerd, etc.) is functioning correctly and able to run containers.	30m	Replace
DiskPressure	DiskPressure is a standard Kubernetes condition indicating the node is experiencing disk pressure (low disk space or high I/O).	N/A	None
KernelReady	KernelReady indicates whether the kernel is functioning correctly without critical errors, panics, or resource exhaustion.	30m	Replace
MemoryPressure	MemoryPressure is a standard Kubernetes condition indicating the node is experiencing memory pressure (low available memory).	N/A	None
NetworkingReady	NetworkingReady indicates whether the node’s networking stack is functioning correctly (interfaces, routing, connectivity).	30m	Replace
StorageReady	StorageReady indicates whether the node’s storage subsystem is functioning correctly (disks, filesystems, I/O).	30m	Replace
Ready	Ready is the standard Kubernetes condition indicating the node is healthy and ready to accept pods.	30m	Replace

By default, EKS automatic node repair stops taking new repair actions when more than 20 percent of the nodes in a group are unhealthy, which limits the impact of a widespread problem on cluster capacity. In-progress repair actions continue. The definition of the group, and the additional scenarios that stop repair, depend on the compute type.

Compute type	Repair stops when
EKS managed node groups	The node group has more than five nodes and more than 20% of the nodes in the node group are unhealthy, or a zonal shift for your cluster triggers through the Application Recovery Controller (ARC).
EKS Auto Mode and Karpenter	More than 20% of the nodes in the NodePool are unhealthy. For standalone NodeClaims, 20% of the nodes in the cluster are unhealthy. Karpenter records a `NodeRepairBlocked` event on the affected nodes with a message such as `more than 20% nodes are unhealthy in the nodepool`.

See Configure automatic node repair for how to override these default settings.

Important

Automatic node repair is a forceful action. On EKS Auto Mode, and on self-managed Karpenter earlier than v1.15.0, repair deletes the Pods on the unhealthy node with a minimal grace period rather than performing a standard drain, and it bypasses Pod disruption budgets. For more information, see Node Auto Repair in the Karpenter documentation. Design workloads that can’t be interrupted safely, such as long-running training jobs, to checkpoint and resume.

Self-managed Karpenter v1.15.0 and later applies a termination grace period and respects Pod disruption budgets within it, and honors the karpenter.sh/do-not-repair annotation. For more information, see Repair behavior on Karpenter v1.15.0 and later.

On every compute type, the karpenter.sh/do-not-disrupt annotation doesn’t exclude a node from repair.

How the repair action is chosen

For most node conditions, the repair action depends only on the condition status: a node whose condition holds the unhealthy status for the wait period is repaired. For AcceleratedHardwareReady, the action also depends on the condition reason, because the EKS node monitoring agent reports the specific GPU or Neuron fault as the reason. For the per-code actions, see NVIDIA XID error codes.

The condition reason can change while a fault persists, because a single fault can be detected through more than one path. For an example, see AcceleratedHardware node health issues. The effect of a change of reason depends on the compute type.

EKS managed node groups

A change of reason can change the outcome, because the repair action follows the reason. Reasons whose default action is None, such as DCGMError and DCGMHealthCode[Code], aren’t repaired. For example, the reason might change from NvidiaXID79Error to DCGMHealthCode120. The default action changes from Replace to None, and the node is no longer repaired. To repair nodes for a reason that isn’t repaired by default, add a nodeRepairConfigOverrides entry for it, as described in Managed node groups.

EKS Auto Mode

A change of reason has no effect. Repair acts on the condition status alone, so the node is replaced after the wait period whatever the reason, including reasons that managed node groups don’t repair.

Self-managed Karpenter v1.15.0 and later

A change of reason can change the wait period, because each repair policy matches on the reason. A reason that no policy matches falls through to a longer wait. For more information, see Repair behavior on Karpenter v1.15.0 and later.

Replacement sequence on EKS Auto Mode and Karpenter

When the repair action is Replace on EKS Auto Mode or Karpenter, the repair follows this sequence:

The node reports an unhealthy node condition.
The wait period for that condition passes.
Karpenter marks the node for deletion and taints it with karpenter.sh/disrupted:NoSchedule. On self-managed Karpenter, the controller logs the repair decision. On EKS Auto Mode, Karpenter runs in the AWS managed control plane, so there are no controller logs to view.
Karpenter launches a replacement NodeClaim. The replacement launches before the unhealthy node is terminated, so both nodes exist for a few minutes.
The unhealthy node is terminated, and Pods reschedule onto the replacement node.

The replacement node is created from the same NodePool, so it can land in any Availability Zone and on any instance type that the NodePool allows.

Repair behavior on Karpenter v1.15.0 and later

Karpenter v1.15.0 runs node repair as a disruption method rather than through the legacy node repair controller, and this is the default in that version and later. The repair actions are the same as the defaults described in this topic, with the exceptions noted in NVIDIA XID error codes, but the following behavior differs. EKS Auto Mode and earlier Karpenter versions don’t include these changes.

Reboot support

Karpenter can reboot a node instead of replacing it when you enable the AWS-specific RebootForRepair feature gate, which is disabled by default. Set it through the AWS_FEATURE_GATES environment variable or the --aws-feature-gates CLI option in the Karpenter deployment. A reboot retains the instance, which avoids reprovisioning accelerated capacity that might have limited availability.

Termination grace period

Repair doesn’t terminate the node immediately. Karpenter applies a termination grace period and respects Pod disruption budgets within it. The following table shows the grace period that each matched repair policy applies.

Matched repair policy	Grace period
An `AcceleratedHardwareReady` reason matched by one of the XID repair policies	5m
Any other condition or reason, including an `AcceleratedHardwareReady` reason that fell through to the 30-minute wait	10m

The selected repair policy log entry reports the period that was applied as termination-grace-period.

The do-not-repair annotation

Setting the karpenter.sh/do-not-repair: "true" annotation on a node excludes it from automatic node repair. The karpenter.sh/do-not-disrupt annotation still doesn’t exclude a node from repair.

To run the legacy node repair controller on these versions instead, set the --legacy-node-repair CLI option or the LEGACY_NODE_REPAIR environment variable to true. The legacy controller only replaces nodes and doesn’t support reboot, termination grace periods, or the karpenter.sh/do-not-repair annotation. The Karpenter project plans to deprecate it.

Configure automatic node repair

Automatic node repair cannot be configured when using EKS Auto Mode. It is always enabled there, with the default settings described in this topic.

Karpenter

To use automatic node repair with Karpenter, enable the feature gate NodeRepair=true. You can enable the feature gates through the --feature-gates CLI option or the FEATURE_GATES environment variable in the Karpenter deployment. For more information, see the Karpenter documentation.

On Karpenter v1.15.0 and later, you can also enable the RebootForRepair feature gate to allow reboots, and annotate nodes with karpenter.sh/do-not-repair to exclude them from repair. For more information, see Repair behavior on Karpenter v1.15.0 and later.

Managed node groups

You can enable automatic node repair when creating new EKS managed node groups or by updating existing EKS managed node groups.

Amazon EKS console – Select the Enable node auto repair checkbox for the managed node group. For more information, see Create a managed node group for your cluster.
AWS CLI – Add --node-repair-config enabled=true to the eks create-nodegroup or eks update-nodegroup-config command.
eksctl – Configure managedNodeGroups.nodeRepairConfig.enabled: true, see the example in the eksctl GitHub.

When using EKS managed node groups, you can control node auto repair behavior with the following settings.

To control when node auto repair stops taking action, set a threshold based on the number of unhealthy nodes in the node group. Set either the absolute count or percentage, but not both.

Setting	Description
`maxUnhealthyNodeThresholdCount`	The absolute number of unhealthy nodes above which node auto repair stops. Use this to limit the scope of repairs.
`maxUnhealthyNodeThresholdPercentage`	The percentage of unhealthy nodes above which node auto repair stops (0-100).

To control how many nodes repair at the same time, you can configure repair parallelism. As with the unhealthy node threshold, set either the absolute count or percentage, but not both.

Setting	Description
`maxParallelNodesRepairedCount`	The maximum number of nodes to repair concurrently.
`maxParallelNodesRepairedPercentage`	The maximum percentage of unhealthy nodes to repair concurrently (0-100).

With nodeRepairConfigOverrides, you can customize repair behavior for specific conditions. Use this setting when different issue types require different repair actions or wait times.

Each override requires all of the following fields:

Field	Description
`nodeMonitoringCondition`	The node condition type reported by the EKS node monitoring agent. For example: `AcceleratedHardwareReady`, `NetworkingReady`, `StorageReady`, `KernelReady`.
`nodeUnhealthyReason`	The specific reason code for the unhealthy condition. For example: `NvidiaXID79Error`, `IPAMDNotRunning`.
`minRepairWaitTimeMins`	The minimum time in minutes that the condition must persist before the node becomes eligible for repair. Use this to avoid repairing nodes for temporary issues. The value must be from 10 through 120 and a multiple of 10.
`repairAction`	The action to take when conditions are met. Valid values: `Replace` (terminate and replace the node), `Reboot` (reboot the node), or `NoAction` (no repair actions).

The following AWS CLI example creates a node group with custom repair settings.


aws eks create-nodegroup \
  --cluster-name my-cluster \
  --nodegroup-name my-nodegroup \
  --node-role arn:aws:iam::111122223333:role/NodeRole \
  --subnets subnet-0123456789abcdef0 \
  --node-repair-config '{
    "enabled": true,
    "maxUnhealthyNodeThresholdPercentage": 10,
    "maxParallelNodesRepairedCount": 3,
    "nodeRepairConfigOverrides": [
      {
        "nodeMonitoringCondition": "AcceleratedHardwareReady",
        "nodeUnhealthyReason": "NvidiaXID46Error",
        "minRepairWaitTimeMins": 10,
        "repairAction": "Replace"
      },
      {
        "nodeMonitoringCondition": "AcceleratedHardwareReady",
        "nodeUnhealthyReason": "NvidiaXID63Error",
        "minRepairWaitTimeMins": 20,
        "repairAction": "NoAction"
      },
      {
        "nodeMonitoringCondition": "AcceleratedHardwareReady",
        "nodeUnhealthyReason": "DCGMHealthCode120",
        "minRepairWaitTimeMins": 10,
        "repairAction": "Replace"
      }
    ]
  }'

This configuration does the following:

Enables node auto repair
Stops repair actions when more than 10% of nodes are unhealthy
Repairs up to 3 nodes at a time
Overrides XID 46 errors (GPU stopped processing) to replace the node after 10 minutes. The default is reboot.
Overrides XID 63 errors (GPU memory remapping event, which is often recoverable) to take no action. The default is reboot.
Repairs nodes whose condition reason is DCGMHealthCode120 (DCGM_FR_FALLEN_OFF_BUS) by replacing them after 10 minutes. The default action for DCGMHealthCode[Code] reasons is none.

Warning Javascript is disabled or is unavailable in your browser.

To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

Node health detection

Configure node monitoring