Help improve this page
To contribute to this user guide, choose the Edit this page on GitHub link that is located in the right pane of every page.
How resource optimization works with autoscalers and workload types
EKS Auto Mode resource optimization recommends CPU and memory requests for your workloads based on the usage it observes for each container, and in Apply mode resizes Pods to match. If you already run autoscalers, Java services, or Pods with sidecars, use this topic to learn what resource optimization changes for those workloads and what it leaves alone.
This topic covers the following sections.
For each workload that a policy selects, resource optimization creates a ResourceOptimization object (short name ro) in the workload’s namespace. Its name is the workload name followed by -deploy for a Deployment or -sts for a StatefulSet. The behavior in this topic appears as conditions and status fields on that object.
Run the following kubectl command to view the object, replacing web-deploy with the name of your workload’s ResourceOptimization object and shop with its namespace:
kubectl get roweb-deploy-nshop-o yaml
Supported workloads
Resource optimization manages Deployments and StatefulSets. It doesn’t manage DaemonSets, Jobs, CronJobs, ReplicaSets that you create directly, Pods without a controller, or custom workload controllers.
Resource optimization never manages workloads in the kube-system, kube-public, or kube-node-lease namespaces, even when a policy selects them.
If a workload’s Pod template sets the isolated-sandbox or confidential-sandbox RuntimeClass, resource optimization recommends for it. It never changes those Pods. A sandboxed Pod’s CPU and memory are fixed when it starts, so it can’t be resized in place. For these workloads, resource optimization doesn’t do any of the following:
-
Resize running Pods
-
Size new Pods
-
Raise memory after an out-of-memory kill
-
Hand back startup CPU
The ResourceOptimization shows the InPlaceResizeUnsupported condition as True with the reason SandboxRuntime. For more information, see How resource optimization applies changes to Pods.
To exclude a single workload, add the optimization.eks.amazonaws.com/enabled: "false" annotation to the Deployment or StatefulSet, not to its Pod template. When you opt out a workload, resource optimization deletes its ResourceOptimization object. It stops making recommendations and cancels any resize that the kubelet hasn’t applied yet. Resizes that the kubelet already applied stay in place, and new Pods start at the size in the Pod template. For the procedure, see Opt out a single workload.
Horizontal Pod Autoscaler and KEDA
Resource optimization doesn’t change the CPU or memory that a Horizontal Pod Autoscaler (HPA) scales on. If resource optimization lowered a request that an HPA measures utilization against, the HPA would read higher utilization and add replicas. The two would then work against each other.
Resource optimization stops managing a resource only when an HPA has a Resource or ContainerResource metric for cpu or memory with a target of type Utilization. It keeps managing the workload’s other resources. The following table shows what resource optimization does for each type of HPA metric.
| HPA metric | What resource optimization does |
|---|---|
|
|
Leaves CPU alone for every container in the Pod. Keeps sizing memory. |
|
|
Leaves memory alone for every container in the Pod, including out-of-memory recovery. Keeps sizing CPU. |
|
|
Leaves that resource alone for the named container only. |
|
|
Keeps sizing the resource, but keeps the container’s limit above the HPA target so that the Pods can still reach it and the HPA can scale out. |
|
|
Keeps sizing both CPU and memory. |
KEDA scales workloads through an HPA that it creates for each ScaledObject, so the same rules apply to KEDA. A ScaledObject that scales on an external event source, such as queue length or consumer lag, leaves CPU and memory to resource optimization. A KEDA cpu or memory trigger with the Utilization metric type doesn’t.
When an HPA owns a resource, the ResourceOptimization shows the HPAResourceConflict condition as True, and status.resourceOwnership lists each resource, container, and HPA. The reason is PartialOwnership when resource optimization still manages some of the workload’s resources, and NoEligibleObjectives when the HPA owns all of them. The Ready condition stays True. Resource optimization still shows recommendations for the HPA-owned resource so that you can review them, but it never applies them.
Run the following kubectl command to view which resources an HPA owns, replacing web-deploy and shop with your values:
kubectl get roweb-deploy-nshop-o jsonpath='{.status.conditions[?(@.type=="HPAResourceConflict")].message}'
An example output is as follows.
cpu (all containers) is scaled by HorizontalPodAutoscaler/web
When you delete the HPA or change it to scale on something else, resource optimization waits for a short stabilization period before it manages the resource again. If KEDA deletes and recreates its HPA, the wait prevents a resize in between.
If resource optimization can’t read the HPAs for a workload, it doesn’t change that workload until it can.
Vertical Pod Autoscaler
Resource optimization detects Vertical Pod Autoscaler (VPA) objects automatically. You don’t need to exclude VPA-managed workloads yourself. How resource optimization responds depends on the VPA’s update mode:
-
Off: The VPA only makes recommendations, so resource optimization manages the workload as usual. -
Initial: Resource optimization doesn’t apply usage-based resizes or startup boost to the workload. TheResourceOptimizationshows theReadycondition asFalsewith the reasonVPAConflict. Because this VPA mode doesn’t change running Pods, resource optimization still raises memory after an out-of-memory kill. -
Any other mode: Resource optimization doesn’t apply usage-based resizes or startup boost, and shows
VPAConflictas forInitial. It also leaves out-of-memory kills to the VPA for the containers whose memory the VPA controls.
To stop resource optimization from tracking a VPA-managed workload at all, add the optimization.eks.amazonaws.com/enabled: "false" annotation to the workload, or leave the workload out of your policy’s selectors. For more information, see How resource optimization policies select and combine.
Out-of-memory recovery
When Kubernetes kills a container because it ran out of memory, the container status shows the reason OOMKilled, and resource optimization raises that container’s memory. In Apply mode, resource optimization applies the raise to the running Pods. In Recommend mode, it only records the raise as a recommendation.
-
Size of the raise: The first kill raises the memory request by 20%. A second kill within 24 hours raises it by 50%, and each further kill within 24 hours doubles it. Resource optimization counts each kill once.
-
Limits: When the container has a memory limit, resource optimization raises the limit by the same ratio, so the container keeps its request-to-limit ratio and its quality of service class. Resource optimization doesn’t add a limit to a container that doesn’t have one.
-
Caps: A raise never goes above the
Maximumthat theBalancedprofile sets, which is twice the largest memory limit in the Pod template. It also never goes above what the largest node in your NodePools can fit. If any container in the Pod has a memory request but no limit, there’s noMaximum, and only the node cap applies. When memory reaches either cap, theResourceOptimizationshows theOOMAtMaximumcondition with the reasonMemoryCapReached. If a workload keeps running out of memory at theMaximum, raise its memory limit in the Pod template or reduce its memory use. If it’s at the node cap, allow larger instance types in a NodePool that the Pod can use, or raise that NodePool’s limits. -
Node memory pressure: Sometimes the kernel kills a container that is below its own limit because the node ran low on memory. If that’s the only kill in 24 hours, resource optimization doesn’t raise the memory.
-
Your guardrails: If one of your own guardrails rejects the raise, the
ResourceOptimizationshows theOOMRaiseRejectedcondition with the reasonRejectedByPolicy. A guardrail can be aResourceQuota, aLimitRange, an admission webhook, or aValidatingAdmissionPolicy. Rejections from webhooks and policies with names that end in.amazonaws.comor.k8s.aws, or that start witheks-orvpc-resource-, don’t set this condition. EKS and AWS use these names for the webhooks and policies that they install, so don’t use them for your own.
Resource optimization needs a memory request or limit on the container to raise it. It doesn’t raise memory that an HPA scales on, or for Pods in a sandbox RuntimeClass. For how raises reach running Pods, see How resource optimization applies changes to Pods.
Memory decreases
You can’t lower a container’s memory limit below the memory that the container is using. The kubelet refuses the resize. A recommendation applies to every Pod of a workload. A busy Pod can use more memory than the new limit. To avoid a resize that the kubelet would refuse, resource optimization doesn’t lower a Pod’s memory if the new limit would be less than 10% above that Pod’s recent peak usage. It still applies the CPU change, and it tries the memory decrease again on a later check. If the kubelet refuses a decrease anyway, resource optimization restores the memory that the Pod is running with.
When any Pod keeps its memory this way, the ResourceOptimization shows the MemoryDecreaseHeld condition as True, with the reason MemoryInUse or MemoryDecreaseRefused. A held Pod can run above the recommendation until its usage drops or it’s replaced.
Java workloads
A Java application reserves heap memory that its measured usage might not show. If resource optimization sized memory from usage alone, the JVM could fail after resource optimization lowered its memory. To prevent this, resource optimization keeps each Java container’s memory request at or above a floor based on the maximum heap. The floor is the larger of 1.25 times the maximum heap, or the maximum heap plus 256 MiB. The extra memory covers metaspace, code cache, and thread stacks. The floor applies to recommendations and to out-of-memory raises.
Resource optimization reads the maximum heap from the container’s spec, in the following order:
-
The
JAVA_TOOL_OPTIONSandJDK_JAVA_OPTIONSenvironment variables -
The options of a
javacommand in the container’scommandandargs -
The
_JAVA_OPTIONSenvironment variable
The last -Xmx or -XX:MaxHeapSize setting wins, and it takes precedence over -XX:MaxRAMPercentage.
The following example Deployment manifest sets a 1 GiB maximum heap, so resource optimization never sets the memory request below 1.25 GiB. Replace public.ecr.aws/amazoncorretto/amazoncorretto:21 with your application’s image, and /app/orders.jar with the path to your application’s JAR file:
apiVersion: apps/v1 kind: Deployment metadata: name: orders namespace: shop spec: replicas: 2 selector: matchLabels: app: orders template: metadata: labels: app: orders spec: containers: - name: orders image:public.ecr.aws/amazoncorretto/amazoncorretto:21command: - java - -Xmx1g - -jar -/app/orders.jarresources: requests: cpu: "1" memory: 2Gi limits: memory: 2Gi
In this preview, heap floors have the following limits:
-
Java only: Resource optimization doesn’t detect heap settings for other runtimes, such as Node.js, Go, or .NET.
-
Literal values only: Resource optimization reads only the values that are written in the Pod template. It can’t see a heap set in any of the following ways:
-
A ConfigMap or Secret (
valueFromorenvFrom) -
A
$(VAR)reference -
An
@argfileor a-XX:VMOptionsFile -
A
javacommand that a shell script startsIn those cases, resource optimization falls back to any
-Xmxthat it can see in the container’s environment variables, command, or arguments. This includesJAVA_OPTS, which the JVM doesn’t read but many startup scripts pass tojava. If it finds none, it sizes the container from usage only, and theResourceOptimizationshows theHeapUndetectedcondition with the reasonNoUnambiguousMaximumHeap. To get a floor, set-Xmxdirectly in the container’s environment variables or arguments.
-
-
Memory limit for percentage heaps:
-XX:MaxRAMPercentagesets the heap as a share of the container’s memory limit. Without a limit, resource optimization can’t know the heap size and applies no floor. With a limit, resource optimization keeps the limit unchanged so that the heap size stays the same. -
Floors above a bound: If the floor is above the
Maximum(twice the largest memory limit in the Pod template), the container’s memory limit, or what the largest node can fit, theResourceOptimizationshows theHeapFloorInfeasiblecondition. The reason names the bound:MaximumBelowFloor,MemoryLimitBelowFloor,NodeCapacityBelowFloor, orMemoryLimitAboveMaximum. To fix it, lower the heap or raise the bound.
How resource settings affect sizing
How you set requests and limits in your Pod template changes what resource optimization can do. The following table shows how resource optimization handles each combination of request and limit settings for a resource.
| Container settings | What resource optimization does |
|---|---|
|
Request and limit |
Sizes the request, and the limit by the same ratio. |
|
Request equal to limit (Guaranteed) |
Keeps the request and limit equal, so the Pod stays Guaranteed. |
|
Limit only |
Kubernetes sets the request equal to the limit when it creates the Pod. Resource optimization sizes the container the same way and keeps the request and limit equal. Your template keeps only the limit, and running Pods show both. |
|
Request only |
Sizes the request and doesn’t add a limit. The request can grow up to what the largest node in your NodePools can fit. |
|
Neither |
Recommends a size from usage but never applies it, in either mode, because a resize can’t change a Pod’s quality of service class. The recommendation’s |
The following workload shapes also affect sizing:
-
Multiple containers: Resource optimization recommends a size for each container from that container’s own usage. The
Balancedprofile’s floor and ceiling are shared across the Pod, from half the smallest container request to twice the largest container limit. If any container has a request but no limit for a resource, the resource has no ceiling. -
Native sidecars: Resource optimization sizes init containers with
restartPolicy: Always, such as service mesh proxies and log shippers, the same way as regular containers, including out-of-memory recovery. -
Init containers: Init containers that run once and exit can’t be resized, so resource optimization leaves them alone.
-
Pod-level resources: If the Pod template sets
spec.resources, resource optimization sizes the Pod as a whole from the Pod’s total usage. An HPA that scales on a resource owns that resource for the whole Pod.
For how resource optimization chooses which workloads to manage, see How resource optimization policies select and combine. For how it applies changes to running and new Pods, see How resource optimization applies changes to Pods. For conditions and what to do about them, see Troubleshoot EKS Auto Mode resource optimization.