View a markdown version of this page

How resource optimization applies changes to Pods - Amazon EKS

Help improve this page

To contribute to this user guide, choose the Edit this page on GitHub link that is located in the right pane of every page.

How resource optimization applies changes to Pods

When a workload’s ResourceOptimization is in Apply mode, EKS Auto Mode resource optimization changes the CPU and memory of the workload’s Pods in two ways:

  • It resizes running Pods in place.

  • It sizes new Pods when they’re created, so they start at the recommended size rather than the size in the Pod template.

In Recommend mode, resource optimization produces recommendations but doesn’t change any Pod.

In both modes, resource optimization never edits a Deployment or StatefulSet. The Pod template keeps the requests and limits you declared, and the values that resource optimization applies exist only on the Pods.

Resize running Pods in place

To resize a running Pod, resource optimization uses the Pod’s resize subresource, the Kubernetes in-place Pod resize API. It changes each container’s CPU and memory requests and limits on the Pod.

  • Pods keep running: A resize doesn’t replace the Pod. Whether a container restarts depends on its resizePolicy. With the Kubernetes default, NotRequired, the container is resized without a restart. A container that sets restartPolicy: RestartContainer for a resource restarts when that resource changes.

  • QoS classes don’t change: Each container keeps its ratio of request to limit. A Guaranteed Pod stays Guaranteed, and a Burstable Pod stays Burstable. A container that declares neither a request nor a limit for a resource is never resized for it; resource optimization only recommends a value. For more information, see How resource optimization works with autoscalers and workload types.

  • Pods are resized gradually: A workload’s Pods are resized a few at a time, not all at once.

  • A change counts as applied only when the kubelet confirms it: In the ResourceOptimization status, a recommendation’s phase moves to InProgress while resource optimization applies it or waits until it can, and to Applied once the Pods report the new size.

GitOps tools see no drift

Because the Pod template doesn’t change, Argo CD, Flux, and other tools that manage your manifests, such as kro, have nothing to revert. Your manifests stay the source of truth for the values you declared.

This also means kubectl get deployment shows the values you declared, not the values your Pods run with. The following command shows what a Pod runs with. Replace my-pod and my-namespace with your values.

kubectl get pod my-pod -n my-namespace \ -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.resources}{"\n"}{end}'

The following command shows the recommendation and whether it’s applied, from the workload’s ResourceOptimization. Its name is the workload’s name followed by -deploy for a Deployment or -sts for a StatefulSet. A very long name is shortened, so if you can’t find it, list them with kubectl get ro --all-namespaces. Replace my-app-deploy and my-namespace with your values.

kubectl get ro my-app-deploy -n my-namespace -o yaml

Applied values stay in place

When you opt a workload out, set its policy to Recommend, or delete the policy that selects it, resource optimization stops making changes to it. Running Pods keep the CPU and memory that resource optimization last applied. Resource optimization cancels any resize that it requested and the kubelet hasn’t applied yet.

To return Pods to the values in the Pod template, restart the workload after you stop. For example, the following command restarts a Deployment. Replace my-app and my-namespace with your values. For a StatefulSet, replace deployment with statefulset.

kubectl rollout restart deployment my-app -n my-namespace

There’s no automatic rollback in this preview. If a change causes a problem, opt the workload out with the optimization.eks.amazonaws.com/enabled: "false" annotation, then restart it. A StatefulSet that uses the OnDelete update strategy doesn’t replace its Pods when you restart it. Delete its Pods instead, and the StatefulSet controller recreates them from the Pod template. For how to opt a workload out, see Opt out a single workload.

When a resize can’t be applied

The kubelet applies a resize only if the Pod’s node has room for it. Kubernetes reports this for each Pod, so one Pod can be unable to resize on its node and the workload’s other Pods still resize normally. Resource optimization reports each case as a condition on the workload’s ResourceOptimization, and in one case an event on the Pod. The following table lists each case, what happens, and what you see.

Situation What happens What you see

The node doesn’t have room for an increase right now. The kubelet reports the resize as Deferred.

The kubelet retries the resize. The Pod keeps running at its current size until the resize fits.

InPlaceInfeasible is True with reason NodeCannotHoldResize and a count of affected Pods.

The node can never hold the increase. The kubelet reports the resize as Infeasible, or the API server rejects it for node capacity.

The Pod keeps its current size. A healthy Pod isn’t evicted to apply a larger size. It picks up the recommendation when it’s next replaced for any other reason.

InPlaceInfeasible is True with a count of affected Pods. Each affected Pod gets a Warning event with reason ResizeInfeasible.

A container keeps getting killed for running out of memory, and the memory increase can’t be applied in place: the kubelet reports it as Infeasible, or as Deferred on a Pod that isn’t Ready.

If new Pods are sized at admission (the AdmissionSizing condition is True), the Pod is evicted through the Eviction API. The replacement Pod starts with the increased memory, and EKS Auto Mode provisions a node that fits it. Pods are never deleted directly. If new Pods aren’t sized at admission, the Pod keeps its current size. A Ready Pod whose increase is Deferred isn’t evicted; it waits for the increase to fit.

The recommendation moves to Applied once the replacement runs at the new size.

A PodDisruptionBudget doesn’t allow that eviction.

The Pod is left running, and the eviction is retried on later passes.

EvictionRefused is True with reason DisruptionBudget.

A ResourceQuota, LimitRange, validating webhook, or ValidatingAdmissionPolicy that you manage rejects a memory increase after an out-of-memory kill.

The increase is retried on later passes.

OOMRaiseRejected is True with reason RejectedByPolicy. The message names what rejected the increase.

A memory decrease would set the limit less than 10% above what the container recently used.

The memory decrease is held, and the rest of the resize is applied. The memory is lowered once usage allows it.

MemoryDecreaseHeld is True with reason MemoryInUse, or MemoryDecreaseRefused if the kubelet refused the decrease.

The Pod template sets the isolated-sandbox or confidential-sandbox RuntimeClass. A sandboxed Pod’s CPU and memory are fixed when it starts, so it can’t be resized in place.

Recommendations are still produced, but none are applied. The Pods aren’t resized, new Pods aren’t sized at admission, and Pods aren’t evicted, including after an out-of-memory kill. To apply a recommendation, set the values in the Pod template yourself.

InPlaceResizeUnsupported is True with reason SandboxRuntime.

The following command lists a workload’s conditions. Replace my-app-deploy and my-namespace with your values.

kubectl get ro my-app-deploy -n my-namespace \ -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

The following command lists the ResizeInfeasible events for Pods whose node can never hold their resize. Replace my-namespace with your namespace.

kubectl get events -n my-namespace --field-selector reason=ResizeInfeasible

How new Pods are sized

New Pods start at the recommendation, not at the size in the Pod template. This covers every new Pod a workload creates. Examples include rollouts, scale-outs from a HorizontalPodAutoscaler or KEDA, and replacements when EKS Auto Mode consolidates or replaces a node. A resource that an HPA holds, or that a container doesn’t declare, still starts at the value in the Pod template.

New Pods are sized at admission by a Kubernetes MutatingAdmissionPolicy named resource-optimization.eks.amazonaws.com, with one MutatingAdmissionPolicyBinding for each workload that resource optimization sizes. A workload has a binding only while it’s in Apply mode and has a recommendation. The policy sizes only the Pods that the workload owns. Other Pods that carry the same labels, such as a Job’s Pods or bare Pods, start at their own size.

The AdmissionSizing condition on the ResourceOptimization says whether new Pods are sized, and why not when they aren’t:

Reason Meaning

Sized

New Pods start at the recommendation.

SizedExceptStartupCPU

The workload uses startup boost. New Pods start at the template’s CPU and at the recommendation for everything else. For more information, see Startup boost.

RecommendMode

The workload is in Recommend mode, so new Pods start at the template.

NoRecommendationYet

The workload doesn’t have a recommendation yet.

SelectorOverlap

Another workload in the namespace selects the same Pods, so a new Pod can’t be matched to one workload.

StartupBoost

The workload uses startup boost, and resource optimization hasn’t recorded its startup boost status yet, so new Pods start at the template.

RecommendOnly

No container in the workload declares a CPU or memory request or limit, so resource optimization only recommends values and new Pods start at the template.

SandboxRuntime

The Pod template sets the isolated-sandbox or confidential-sandbox RuntimeClass, so new Pods start at the template.

NotReady

The ResourceOptimization isn’t Ready, for example because a VerticalPodAutoscaler also targets the workload (Ready is False with reason VPAConflict), so new Pods start at the template.

UnsupportedWorkloadKind

The workload isn’t a Deployment or StatefulSet, so new Pods start at the template.

WorkloadSelectorUnavailable

The target workload doesn’t exist or its Pod selector is empty, so new Pods start at the template.

AdmissionSizingUnavailable

The cluster’s API server doesn’t serve MutatingAdmissionPolicy, so new Pods start at the template.

ReconcileFailed, AdmissionSizingNotPermitted, or AdmissionPolicyMissing

Resource optimization couldn’t set up sizing for the workload, so new Pods start at the template. The message says why, and resource optimization retries.

When you opt a workload out, set its policy to Recommend, or delete its policy, the workload’s binding is removed. From then on, new Pods start at the size in the Pod template.

Important

In this preview, resource optimization doesn’t take LimitRange objects into account. If a recommendation doesn’t meet a LimitRange in the namespace (its maximum, minimum, or maxLimitRequestRatio), the API server rejects new Pods that are sized to it. If you use LimitRanges, make sure they allow for the recommendations, or opt the affected workloads out.

Startup boost

Some applications, such as Java services, need much more CPU while they start than while they run. Startup boost gives a workload extra CPU for startup, and hands it back after each Pod is Ready.

You set the startup CPU yourself, as the container’s CPU request in the Pod template. Resource optimization doesn’t add CPU. The boost lasts until the Pod is Ready; there’s no timer and no multiplier.

Use startup boost when a workload needs noticeably more CPU to start than to run, and a smaller request would make it start slowly or fail its probes. If your workload doesn’t need extra CPU to start, don’t use it. Without startup boost, a raised CPU request comes down only to the Balanced floor, half of the declared request, and new Pods start without the extra CPU.

How startup boost works

  1. You raise the container’s CPU request in the Pod template to what it needs for startup, and add the optimization.eks.amazonaws.com/startup-boost: "true" annotation to the workload.

  2. A new Pod starts at the template’s CPU. Its memory, and the resources of its other containers, are sized at admission as usual.

  3. During startup, the Pod’s CPU is left alone.

  4. After the Pod is Ready, its CPU is resized in place down to the steady-state recommendation.

Startup boost also changes two other things for the workload:

  • The CPU floor is waived: The Balanced profile doesn’t normally lower a request below half of the declared request. With startup boost, the declared CPU is the startup value, so the floor is waived for CPU and the request can go as low as the recommendation. The CPU recommendation still has a minimum of 10 millicores. Memory keeps the normal floor. The CPUFloorWaived condition is True while the waiver applies.

  • Startup CPU doesn’t count toward the recommendation: CPU that a Pod uses while it starts is left out of the usage that recommendations come from, so startup spikes don’t raise the steady-state recommendation. Memory used during startup still counts.

Prerequisites

Startup boost hands back CPU only when all of the following are true. When one isn’t, Pods keep the template’s CPU, and the StartupShrinkBlocked condition gives the reason. The following table lists each requirement and the reason shown when it isn’t met.

Requirement Reason when it isn’t met

The workload is in Apply mode.

RecommendMode

There’s a steady-state CPU recommendation for the container, which takes about 12 hours of usage history. On a new workload, Pods keep the startup CPU until then.

SteadyRecommendationUnavailable

The workload doesn’t use Pod-level resources (resources in the Pod spec).

PodLevelResources

No HorizontalPodAutoscaler scales the workload on CPU utilization. Resource optimization doesn’t change a CPU request that an HPA’s target depends on.

CPUScaledByHPA

No VerticalPodAutoscaler in a mode other than Off targets the workload.

VPAConflict

The Pod template doesn’t set the isolated-sandbox or confidential-sandbox RuntimeClass.

SandboxRuntime

The kubelet can resize the Pod in place on its node.

PodResizeBlocked. The message gives the number of Pods that couldn’t be resized and keep their startup CPU.

Give each container a readiness probe or a startup probe that passes only once the application has started, because a Pod’s startup is treated as over when it’s Ready. A Pod without either probe is Ready as soon as its containers start, so for such a Pod the CPU is handed back 10 minutes after the Pod was created instead.

Turn on startup boost

The following YAML example declares 2 CPUs for startup on a Java service that needs about 400 millicores once it’s running. Its readiness probe passes only after the application has started. Replace the image and the probe path with your own.

apiVersion: apps/v1 kind: Deployment metadata: name: orders namespace: shop annotations: optimization.eks.amazonaws.com/startup-boost: "true" spec: replicas: 3 selector: matchLabels: app: orders template: metadata: labels: app: orders spec: containers: - name: orders image: 111122223333.dkr.ecr.us-west-2.amazonaws.com/orders:1.0 ports: - containerPort: 8080 resources: requests: cpu: "2" memory: 1Gi limits: memory: 1Gi readinessProbe: httpGet: path: /ready port: 8080 periodSeconds: 5

The annotation goes on the Deployment or StatefulSet itself, not on the Pod template. To turn startup boost off, remove the annotation.

Check startup boost

The following command shows the startupShrink field in the ResourceOptimization status, which counts the workload’s Ready Pods:

kubectl get ro orders-deploy -n shop -o jsonpath='{.status.startupShrink}'

An example output is as follows.

{"blockedPods":0,"eligiblePods":3,"matchingPods":3,"pendingPods":0}
  • eligiblePods: Ready Pods whose startup CPU is to be handed back. It equals matchingPods plus pendingPods plus blockedPods.

  • matchingPods: Pods already at their steady-state CPU

  • pendingPods: Pods still being resized to their steady-state CPU. The StartupShrinkPending condition is True while there are any.

  • blockedPods: Pods that keep the startup CPU because they can’t be resized. The StartupShrinkBlocked condition gives the reason.

Startup boost and node autoscaling

For each container whose startup CPU resource optimization hands back, the workload’s ResourceOptimization lists the startup CPU from the Pod template in status.newPodSize. When EKS Auto Mode moves the workload’s Pods off a node, for example to consolidate or replace nodes, it plans capacity for those Pods with this CPU. It doesn’t use the lower CPU that the running Pods use after startup.