Help improve this page
To contribute to this user guide, choose the Edit this page on GitHub link that is located in the right pane of every page.
Troubleshoot EKS Auto Mode resource optimization
Use kubectl to find out why EKS Auto Mode resource optimization isn’t doing what you expect. This topic covers how to stop changes during an incident. It also covers the common reasons a workload isn’t managed, has no recommendation, or doesn’t get a recommendation applied. If you need to stop changes now, see Stop changes and revert values during an incident.
Resource optimization reports what it’s doing in the status of two kinds of objects:
-
A
ClusterResourceOptimizationPolicy(crop) reports whether the policy works. It also reports how many workloads it selects. -
A
ResourceOptimization(ro) for each managed workload reports its recommendations and conditions. The object is in the workload’s namespace, and its name is the workload’s name with a-deployor-stssuffix.
In the commands in this topic, replace the following placeholders:
-
my-app: the name of your Deployment or StatefulSet. -
my-namespace: the workload’s namespace. -
my-policy: the name of your policy. -
my-pod: the name of one of the workload’s Pods.
The following kubectl commands show most of what you need:
kubectl get crop kubectl get ro -nmy-namespacekubectl describe romy-app-deploy-nmy-namespacekubectl get romy-app-deploy-nmy-namespace-o yaml
The Conditions section of kubectl describe ro lists each condition with its status, reason, and message. The message usually says how many Pods are affected and why. For every condition and reason, see EKS Auto Mode resource optimization resource reference.
Stop changes and revert values during an incident
Resource optimization has no automatic rollback. If you think resource changes are causing a problem, first stop changes, then revert the values.
Stop changes
Use one of the following options. Each option stops new resizes for the workloads it covers. It also cancels any resize that the kubelet hasn’t applied yet. Values that the kubelet already applied stay in place.
-
Stop changes for one workload. Run the following command to add the opt-out annotation to the Deployment. For a StatefulSet, replace
deploymentwithstatefulset.kubectl annotate deploymentmy-app-nmy-namespaceoptimization.eks.amazonaws.com/enabled="false"Resource optimization deletes the workload’s
ResourceOptimizationobject and its recommendations. New Pods start at the values in the Pod template. This also deletes the workload’s usage history. If you opt the workload back in, the wait of about 12 hours for a first recommendation starts over. For more information, see Opt out a single workload. -
Stop changes but keep recommendations. Set
mode: Recommendon the policy that manages the workload. ThePOLICYcolumn ofkubectl get roshows which policy that is. Run the following command:kubectl patch cropmy-policy--type merge -p '{"spec":{"mode":"Recommend"}}'Running Pods keep their current values, new Pods start at the values in the Pod template, and recommendations and usage history are kept. For more information, see Stop resizing but keep recommendations.
-
Turn off resource optimization for the whole cluster. Run the following command to delete every policy. This stops data collection, recommendations, and resizes. Resource optimization deletes every
ResourceOptimizationobject and the usage history of every workload. If you turn resource optimization back on, the wait for first recommendations starts over. For more information, see Turn off resource optimization.kubectl delete crop --all
Revert to the values in the Pod template
Resource optimization never changes your Deployment or StatefulSet. Its Pod template still has the requests and limits you wrote. After you stop changes, run the following command to restart the workload and return its Pods to those values:
kubectl rollout restart deploymentmy-app-nmy-namespace
A StatefulSet that uses the OnDelete update strategy doesn’t replace its Pods when you restart it. Delete its Pods instead. The StatefulSet controller recreates them from the Pod template.
Outside Apply mode, resource optimization doesn’t size new Pods at admission, so they start at the values in the template. To confirm, run the following command to check the resources of a new Pod:
kubectl get podmy-pod-nmy-namespace-o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources}{"\n"}{end}'
Restart the workload only after you stop changes. If the workload is still in Apply mode, new Pods start at the recommended values.
Workload isn’t managed
Symptom:
kubectl get ro -n my-namespace doesn’t list an object for the workload.
Check the following causes in order:
-
No policy exists.
kubectl get cropreturnsNo resources found. Create a policy. For more information, see Turn on resource optimization with default settings. -
The workload is in a system namespace. Resource optimization never manages workloads in
kube-system,kube-public, orkube-node-lease. -
The workload isn’t a Deployment or StatefulSet. Resource optimization doesn’t manage DaemonSets, Jobs, CronJobs, or bare Pods.
-
The workload opts out. Check for the annotation.
kubectl get deploymentmy-app-nmy-namespace-o jsonpath='{.metadata.annotations.optimization\.eks\.amazonaws\.com/enabled}'If the output is
false, remove the annotation to opt the workload back in. -
No policy selects the workload. Compare each policy’s selectors with the labels on the namespace and on the workload object.
kubectl get namespacemy-namespace--show-labels kubectl get deploymentmy-app-nmy-namespace--show-labels kubectl get crop -o yamlworkloadSelectormatches the labels on the Deployment or StatefulSet object, not the labels in its Pod template. This is a common reason a policy doesn’t select a workload. Add the label to the workload’smetadata.labels, or change the selector. -
The policy isn’t Ready, or didn’t sync the workload. See Policy isn’t Ready, or some workloads didn’t sync.
Policy isn’t Ready, or some workloads didn’t sync
Symptom: The READY column of kubectl get crop is False, or the policy’s WorkloadsSynced condition is False.
Run kubectl describe crop my-policy and check the conditions. WorkloadsSynced doesn’t affect Ready, so the policy keeps managing the workloads that did sync. The following table lists the conditions, their causes, and what to do.
| Condition and reason | Cause | Action |
|---|---|---|
|
|
|
Fix the selector. Valid operators are |
|
|
The workload already has a |
Delete the hand-written object. The policy then creates its own object for the workload. To control which workloads resource optimization manages, use a policy’s |
|
|
A temporary error. |
Resource optimization retries automatically. If the condition stays, contact AWS Support. |
There’s no recommendation yet
Symptom: The ResourceOptimization object exists, but status.recommendations is empty. In Apply mode, the AdmissionSizing condition is False with reason NoRecommendationYet.
Cause: Resource optimization makes the first recommendation for a container about 12 hours after it collects the container’s first usage sample. Deleting the workload’s ResourceOptimization object deletes its usage history, and the wait starts over. That happens when you opt the workload out, when no valid policy selects it any longer, or when you delete the last policy that selects it.
Action: Wait. Changing the policy’s mode doesn’t reset the wait or produce a recommendation sooner.
Recommendation wasn’t applied
Symptom: The object has a recommendation, but running Pods still have their old requests.
Run the following commands to check the phase of each recommendation and the conditions. Then find what you see in the table that follows.
kubectl get romy-app-deploy-nmy-namespace-o jsonpath='{range .status.recommendations[].verticalRecommendations[]}{.containerName}{"\t"}{.phase}{"\n"}{end}' kubectl describe romy-app-deploy-nmy-namespace
| What you see | Cause | Action |
|---|---|---|
|
The |
The workload is in |
To apply recommendations, set |
|
|
A VerticalPodAutoscaler that isn’t in |
Choose one tool for the workload. Set the VerticalPodAutoscaler’s |
|
|
A HorizontalPodAutoscaler scales on the utilization of that resource. See Resource is left alone because of a HorizontalPodAutoscaler. |
None. This is expected. |
|
|
The Pods run in a Nitro sandbox, under the |
None. Resource optimization keeps publishing recommendations. To use them, set the values in the Pod template yourself. |
|
The phase is |
The container declares neither a request nor a limit for the resource. Adding a request in place would change the Pod’s quality of service class, so resource optimization only recommends a value. |
To apply the value, add the request to the Pod template yourself. |
|
|
The Pod’s node doesn’t have room for the larger request. If the kubelet reports the resize as |
None. When the Pod is replaced, for example during a rollout, scale-out, or node consolidation, the new Pod starts at the recommended size on a node that fits. To replace the Pods sooner, restart the workload. |
|
|
The recommended memory limit would be less than 10% above the Pod’s recent memory usage, or the kubelet refused the decrease. Resource optimization applies the rest of the resize and keeps the current memory. |
None. Resource optimization lowers memory when usage drops. If usage stays high, the next recommendation accounts for it. |
|
|
The recommendation is larger than the nodes your NodePools can launch, so resource optimization capped it. |
Allow larger instance types in your NodePools, or accept the capped value. |
|
The phase is |
The kubelet hasn’t finished the resize, or resource optimization is waiting before it sends one. For example, resource optimization waits while it can’t read the workload’s HorizontalPodAutoscalers or VerticalPodAutoscalers, because it doesn’t change a workload until it can confirm that no autoscaler holds a resource. No condition reports this wait. |
Run |
New Pods don’t start at the recommended size
Symptom: In Apply mode, new Pods start at the values in the Pod template, and resource optimization then resizes them.
Cause: Resource optimization sizes new Pods at admission only when the AdmissionSizing condition is True. Run the following command to check its reason. Then find the reason in the table that follows.
kubectl get romy-app-deploy-nmy-namespace-o jsonpath='{.status.conditions[?(@.type=="AdmissionSizing")]}'
| Reason | Cause and action |
|---|---|
|
|
The workload is in |
|
|
There’s no recommendation yet. See There’s no recommendation yet. |
|
|
The |
|
|
The workload has the |
|
|
The workload’s containers declare neither a request nor a limit for CPU or memory, so there’s nothing to set at admission. Add requests to the Pod template if you want resource optimization to size new Pods. |
|
|
The Pods run in a Nitro sandbox, which doesn’t support resizing. Resource optimization doesn’t size them. Expected. |
|
|
The workload’s Pod label selector overlaps the selector of another workload that resource optimization manages. While they overlap, resource optimization doesn’t size either workload’s new Pods at admission. Give each workload a distinct selector. |
|
|
Resource optimization couldn’t find the workload or its Pod label selector. Check that the Deployment or StatefulSet named in the object’s |
|
|
The cluster doesn’t serve the |
|
|
A problem on the AWS side. Resource optimization retries automatically. If the condition stays, contact AWS Support. |
Resource optimization doesn’t take LimitRange objects into account when it sizes new Pods. If a namespace has a LimitRange, the API server can reject new Pods that are sized to a recommendation. For example, the recommendation can exceed the LimitRange maximum, fall below its minimum, or break its maxLimitRequestRatio. Change the LimitRange to allow the recommendations, or opt the workload out.
Workload keeps running out of memory
In Apply mode, when the kubelet kills a container for running out of memory, resource optimization raises its memory. It raises memory 1.2 times on the first kill, 1.5 times on the second, and 2 times from the third. If a Pod’s node can’t hold the increase, resource optimization evicts the Pod so that its replacement starts at the new size. It evicts Pods only while new Pods are sized at admission, that is, while the AdmissionSizing condition is True.
If the container keeps getting killed, run kubectl describe ro my-app-deploy -n my-namespace and find the condition in the following table. For Java workloads, also check HeapUndetected and HeapFloorInfeasible, and see How resource optimization works with autoscalers and workload types.
| Condition and reason | Cause | Action |
|---|---|---|
|
|
Memory is already at the |
Raise the memory limit in the Pod template. Resource optimization recalculates the |
|
|
Memory is already as large as the largest node the Pod can run on. |
Allow larger instance types in your NodePools, or reduce the workload’s memory use. |
|
|
A ResourceQuota, LimitRange, validating webhook, or ValidatingAdmissionPolicy in your cluster rejected the increase. The message names it and the resources the increase asked for. |
Change the quota, limit, or policy to allow the increase. Resource optimization retries automatically. |
|
|
The node can’t hold the increase, and new Pods aren’t sized at admission, so a replacement would start at the template size. Resource optimization doesn’t evict the Pod, and it keeps its size. |
Fix the |
|
|
A PodDisruptionBudget doesn’t allow the eviction. Resource optimization never deletes Pods, so the Pod waits. |
Allow the eviction. If the budget is already breached because Pods are crashing, set |
|
|
A HorizontalPodAutoscaler scales on memory utilization. Resource optimization doesn’t raise memory that a HorizontalPodAutoscaler scales on. |
Raise the memory request in the Pod template, or scale on a different metric. |
Resource is left alone because of a HorizontalPodAutoscaler
Symptom: Resource optimization recommends a value for CPU or memory but never applies it. The HPAResourceConflict condition is True, and status.resourceOwnership.vertical lists the resource and the HorizontalPodAutoscaler.
Cause: This is expected. A HorizontalPodAutoscaler that scales on the utilization of a resource calculates utilization from its request. If resource optimization changed that request, it would change when the HorizontalPodAutoscaler scales. Resource optimization leaves the resource alone and still sizes the other one. For example, with a HorizontalPodAutoscaler on CPU utilization, resource optimization still sizes memory. KEDA works the same way through the HorizontalPodAutoscaler it creates.
Action: None. Ready stays True. For more information, see How resource optimization works with autoscalers and workload types.
Startup boost doesn’t return CPU
Symptom: A workload has the optimization.eks.amazonaws.com/startup-boost: "true" annotation, but its Pods keep the raised startup CPU after they’re Ready.
Run the following command to check the StartupShrinkBlocked condition and status.startupShrink. Then find the reason in the table that follows.
kubectl get romy-app-deploy-nmy-namespace-o jsonpath='{.status.startupShrink}{"\n"}{.status.conditions[?(@.type=="StartupShrinkBlocked")]}'
| Reason | Cause and action |
|---|---|
|
|
Startup boost works only in |
|
|
Resource optimization doesn’t have a steady-state CPU recommendation for the container yet, which takes about 12 hours of data. Wait. |
|
|
A HorizontalPodAutoscaler scales the workload on CPU, so resource optimization doesn’t change CPU. You can’t use startup boost with it. |
|
|
The Pod template uses Pod-level resources, which startup boost doesn’t support. |
|
|
A VerticalPodAutoscaler that isn’t in |
|
|
The Pods run in a Nitro sandbox, which doesn’t support in-place resizes. You can’t use startup boost with it. |
|
|
The kubelet reported the resize as |
A Pod’s startup ends when it’s Ready. If the Pod has no readiness or startup probe, it’s Ready as soon as its containers start. Resource optimization then waits 10 minutes from Pod creation before it returns the CPU. Add a readiness or startup probe that passes only when startup is done.