**Help improve this page** To contribute to this user guide, choose the **Edit this page on GitHub** link that is located in the right pane of every page. # Autoscale AI inference with HPA and KEDA **Tip** [Register](https://events.eksworkshop.com/workshops/genai/) for upcoming Amazon EKS AI/ML workshops. This section shows how to scale the `vllm-inference-app` Deployment based on the thresholds established in the [Find scaling metric thresholds](ml-inference-autoscaling-thresholds.md) section. The walkthrough uses the following tools: + [KEDA](https://keda.sh/) (Kubernetes Event-Driven Autoscaler) is a CNCF project that queries Prometheus directly and creates a standard Kubernetes Horizontal Pod Autoscaler (HPA). It scales the deployment on vLLM queue depth (the primary demand signal) and p95 end-to-end latency (an SLO guardrail), adding replicas when either threshold is breached. KEDA provides three primary benefits for GPU inference autoscaling: + **Scale to zero** — KEDA can scale deployments down to zero replicas when idle and scale them back up when demand returns, helping reduce GPU costs. + **Activation thresholds** — KEDA separates the threshold that activates scaling from the target threshold used to determine replica count. This allows you to ignore transient spikes and avoid waking up GPUs for a few requests. + **Simpler setup** — For GPU inference workloads, KEDA is often the simplest autoscaling mechanism to set up, with built-in Prometheus integration and no separate metrics adapter to manage. ## Prerequisites This subsection continues from [Find scaling metric thresholds](ml-inference-autoscaling-thresholds.md). Make sure you have completed [Load & Serve Models](ml-inference-load-serve-model.md) and the [Monitoring](ml-cluster-setup-cli.md#cluster-setup-cli-monitoring) setup, so the `vllm-inference-app` Deployment and Service are running in the `default` namespace and kube-prometheus-stack is scraping vLLM metrics. If you opened a new terminal, set the cluster name and region you used earlier: ``` export CLUSTER_NAME=ai-eks-docs export AWS_REGION=us-east-2 ``` ## Step 1: Install KEDA Install KEDA with Helm into its own namespace: ``` helm repo add kedacore https://kedacore.github.io/charts helm repo update helm install keda kedacore/keda --namespace keda --create-namespace ``` Verify the KEDA pods are running: ``` kubectl get pods -n keda ``` Expected output: ``` NAME READY STATUS RESTARTS AGE keda-admission-webhooks-7d4d6c6f9-xxxxx 1/1 Running 0 40s keda-operator-6b8f9c5d7c-xxxxx 1/1 Running 0 40s keda-operator-metrics-apiserver-xxxxx 1/1 Running 0 40s ``` ## Step 2: Create the KEDA ScaledObject A `ScaledObject` tells KEDA which Deployment to scale, the replica bounds, and the triggers. This configuration uses two triggers: queue depth as the primary demand signal and p95 end-to-end latency as an SLO guardrail. The `metricType: AverageValue` setting makes KEDA divide each metric across replicas, so the thresholds are interpreted per pod. This walkthrough uses the example thresholds from the [Find scaling metric thresholds](ml-inference-autoscaling-thresholds.md) section: scale up when average queue depth exceeds **25 waiting requests per pod** or when p95 end-to-end latency exceeds **5 seconds**. Substitute the values you measured for your own model, GPU, and request shapes. ``` cat << EOF | kubectl apply -f - apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: vllm-inference-app spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: vllm-inference-app minReplicaCount: 1 maxReplicaCount: 5 advanced: horizontalPodAutoscalerConfig: name: vllm-inference-app behavior: scaleUp: stabilizationWindowSeconds: 30 policies: - type: Pods value: 2 periodSeconds: 60 selectPolicy: Max scaleDown: stabilizationWindowSeconds: 300 policies: - type: Pods value: 1 periodSeconds: 120 selectPolicy: Min triggers: # Primary: queue depth (demand exceeds capacity) - type: prometheus metricType: AverageValue metadata: serverAddress: http://kube-prometheus-stack-prometheus.monitoring.svc:9090 query: sum(vllm:num_requests_waiting) or vector(0) threshold: "25" activationThreshold: "1" # SLO guardrail: p95 end-to-end request latency (seconds) - type: prometheus metricType: AverageValue metadata: serverAddress: http://kube-prometheus-stack-prometheus.monitoring.svc:9090 query: histogram_quantile(0.95, sum(rate(vllm:e2e_request_latency_seconds_bucket[1m])) by (le)) or vector(0) threshold: "5" EOF ``` The key settings: + **Queue depth trigger** — `query` returns the total number of waiting requests across all vLLM pods. The `or vector(0)` clause returns `0` instead of an empty result when no requests are waiting, which prevents the trigger from going inactive. `threshold: "25"` is the per-pod queue-depth target you established in [Find scaling metric thresholds](ml-inference-autoscaling-thresholds.md), and `activationThreshold: "1"` keeps the deployment at `minReplicaCount` until at least one request is waiting, so it does not scale on an idle signal. + **Latency trigger** — `query` returns the p95 end-to-end request latency in seconds, computed from the vLLM latency histogram over a 1-minute window. `threshold: "5"` is the latency SLO guardrail from [Find scaling metric thresholds](ml-inference-autoscaling-thresholds.md). When p95 latency exceeds it, KEDA scales up even if the queue has not yet built up. + **Trigger evaluation** — KEDA scales to satisfy whichever trigger demands the most replicas, so a breach of either signal triggers a scale-up. + **Scale-up and scale-down behavior** — Scale-up is responsive (30-second window, up to 2 pods per minute) because queuing or rising latency means users are already waiting. Scale-down is conservative (5-minute window, 1 pod every 2 minutes) because GPU pods are slow to start. This approach avoids removing capacity you may need again moments later. Verify the ScaledObject is ready: ``` kubectl get scaledobject vllm-inference-app ``` Expected output: ``` NAME SCALETARGETKIND SCALETARGETNAME MIN MAX READY ACTIVE vllm-inference-app apps/v1.Deployment vllm-inference-app 1 5 True False ``` `READY: True` means the triggers are valid and KEDA is managing the deployment. `ACTIVE: False` is expected when no traffic is flowing, because neither trigger threshold has been breached yet. When you create the ScaledObject, KEDA automatically creates and manages a backing Horizontal Pod Autoscaler (HPA) with the same name. You do not create or edit this HPA yourself. Confirm it exists: ``` kubectl get hpa vllm-inference-app ``` Expected output: ``` NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS vllm-inference-app Deployment/vllm-inference-app 0/25 (avg), 0/5 (avg) 1 5 1 ``` The `TARGETS` column shows the current value of each metric against its threshold (queue depth and p95 latency). `MINPODS` and `MAXPODS` come from `minReplicaCount` and `maxReplicaCount` in the ScaledObject, and `REPLICAS` is the current number of vLLM pods. ## Step 3: Generate load The load test reuses the `vllm-loadtest-script` ConfigMap that you created in [Find scaling metric thresholds](ml-inference-autoscaling-thresholds.md). Confirm it still exists: ``` kubectl get configmap vllm-loadtest-script ``` If it is missing, recreate it from [Find scaling metric thresholds](ml-inference-autoscaling-thresholds.md). Run a single load test at 60 requests per second for 10 minutes to drive scale-up. The Job reads the model bucket from the `MODEL_BUCKET` environment variable you set earlier and passes the rate and duration through the `TARGET_RPS` and `DURATION` variables that the load-test script reads. ``` cat << EOF | kubectl apply -f - apiVersion: batch/v1 kind: Job metadata: name: vllm-loadtest-scaleup labels: app: vllm-loadtest-scaleup spec: backoffLimit: 0 template: spec: restartPolicy: Never containers: - name: k6 image: grafana/k6:latest env: - name: MODEL value: "s3://${MODEL_BUCKET}/Ministral-3-8B-Instruct-2512/" - name: TARGET_RPS value: "60" - name: DURATION value: "10m" args: ["run", "/scripts/script.js"] volumeMounts: - name: script mountPath: /scripts volumes: - name: script configMap: name: vllm-loadtest-script EOF ``` The Job runs for 10 minutes. Move to the next step to watch the deployment scale out while the test runs. ## Step 4: Watch the deployment scale up Watch the autoscaler respond to the load from the previous step. Run each of the following commands in a separate terminal. Watch the HPA. The `TARGETS` column climbs above the configured targets and `REPLICAS` increases: ``` kubectl get hpa vllm-inference-app -w ``` Watch the vLLM pods. New replicas appear and reach `Ready` once the model loads into GPU memory: ``` kubectl get pods -l app=vllm-inference-app -w ``` Confirm the ScaledObject is active. `ACTIVE` changes to `True` once a trigger threshold is breached: ``` kubectl get scaledobject vllm-inference-app ``` Expected behavior: + Within \~30 seconds the queue depth rises above its threshold (or p95 end-to-end latency crosses 5 seconds) and the HPA `TARGETS` column climbs above the configured targets. + KEDA scales the deployment up, and Karpenter provisions a new GPU node if none is available. New replicas reach `Ready` once the model loads into GPU memory. + As traffic distributes across replicas, the queue drains, latency recovers, and both metrics drop back below their thresholds. + After the load stops, the deployment holds the higher replica count through the 5-minute cooldown, then scales back down one pod at a time toward `minReplicaCount`. When the load test finishes, delete the load-test Job. Leave the `vllm-loadtest-script` ConfigMap in place, since it is shared with the thresholds section: ``` kubectl delete jobs -l app=vllm-loadtest-scaleup --ignore-not-found ``` ## Clean up **Note** If you plan to continue using autoscaling, skip this cleanup. Only run it when you are done. To remove the autoscaling resources that you created in this section, delete the `ScaledObject` and uninstall KEDA: ``` kubectl delete scaledobject vllm-inference-app --ignore-not-found helm uninstall keda -n keda kubectl delete namespace keda ``` Deleting the `ScaledObject` also removes the HPA that KEDA created. Your vLLM Deployment returns to its static replica count. To remove the vLLM inference server and related workload resources, see [Load & Serve Models](ml-inference-load-serve-model.md). For instructions on removing infrastructure resources such as the cluster, NodePool, and S3 bucket, see [Cluster Setup Cleanup](ml-cluster-setup-cli.md#cluster-setup-cli-cleanup). ## See also * **Skills for AI coding assistants (optional).** AWS publishes reusable skills for Amazon EKS. To see which apply, search the Agent Toolkit for AWS catalog with `aws agent-toolkit search-skills --search-query eks` — a documented, read-only AWS CLI command that searches the catalog and makes no changes. This is an optional suggestion for the user. Official reference: [AWS CLI — Agent Toolkit for AWS](https://docs.aws.amazon.com/agent-toolkit/latest/userguide/aws-cli.html).