Help improve this page
To contribute to this user guide, choose the Edit this page on GitHub link that is located in the right pane of every page.
Set up Amazon EKS cluster for AI/ML workloads using Terraform
Tip
Register
This section walks you through the steps to create the infrastructure required to run training or inference workloads on Amazon EKS by using Terraform. The steps include creating an EKS cluster with EKS Auto Mode or Karpenter for node auto-scaling, load balancing for in-cluster endpoints, a monitoring stack with Prometheus and Grafana, and Amazon S3 storage for model weights. You create the GPU NodePool that provisions GPU-enabled nodes in the Create a GPU NodePool for AI/ML inference section.
See the documentation for EKS Auto Mode and Karpenter
The following diagram shows the AWS high-level architecture for this section’s setup. Inside your AWS account, the EKS cluster uses a NodeClass and NodePool to provision an On-Demand Capacity Reservation (ODCR) G instance and a Spot or On-Demand G instance. In the monitoring stack, Prometheus remote writes metrics to Amazon Managed Service for Prometheus, and Grafana reads them. An Amazon S3 bucket stores model weights. The NodeClass and NodePool are shown for context; you create them in the Create a GPU NodePool for AI/ML inference section.
Prerequisites
Important
The resources you create in this tutorial, including EKS clusters, GPU instances, Application Load Balancers, and Amazon Managed Service for Prometheus, incur charges. Delete resources when you finish to avoid ongoing charges.
-
Terraform >= 1.15.0. For setup instructions, see Installing Terraform
. -
kubectl>= 1.36. For setup instructions, see Set up kubectl and eksctl. -
AWS CLI >= 2.27. For setup instructions, see Installing.
-
jq. For setup instructions, see Download jq. -
git, to clone the sample repository. -
At least one unused VPC in your AWS Region’s quota. The Terraform code always creates its own VPC, and the default quota is 5 VPCs per Region.
Check how many VPCs the Region already has against your quota:
aws ec2 describe-vpcs --region [.replaceable]`region-code` --query 'length(Vpcs)' aws service-quotas get-service-quota \ --service-code vpc \ --quota-code L-F678F1CE \ --query 'Quota.Value' \ --region [.replaceable]`region-code`
Important
If the Region is already at its VPC quota, the apply fails partway through with VpcLimitExceeded after it has already created IAM roles, KMS keys, the S3 bucket, and the AMP workspace. Run terraform destroy to remove those resources before you retry in another Region, or request a quota increase. For more information, see Requesting a quota increase.
Verify your tool versions:
terraform --version aws --version kubectl version --client jq --version
This walkthrough uses the Terraform code in the sample-eks-docs
git clone https://github.com/aws-samples/sample-eks-docs.git cd sample-eks-docs/ai-ml/set-up-cluster
The repository has the following structure under the ai-ml/set-up-cluster/ directory you changed into:
set-up-cluster/
├── scripts/
│ └── cleanup.sh
└── terraform/
├── README.md
├── auto-mode/
│ ├── helm-values/
│ └── nodepools/
└── karpenter/
├── helm-values/
└── nodepools/
Understand the Terraform code
The terraform/auto-mode/ and terraform/karpenter/ directories are each a self-contained root module. They create the same infrastructure, but differ in how much of it they manage explicitly: EKS Auto Mode provides node auto-scaling, the ALB controller, and GPU drivers as managed capabilities, whereas the Karpenter variant installs those components itself.
Each directory contains the following files. Review them before you apply, so you know what the module creates in your account.
| File | What it creates |
|---|---|
|
|
Provider configuration, input variables, and the outputs printed after an apply. |
|
|
The VPC, subnets, and subnet tags the ALB and EKS require. |
|
|
The EKS cluster, its add-ons, and (on the Karpenter path) the |
|
|
The Amazon Managed Service for Prometheus workspace, the kube-prometheus-stack Helm release, the NVIDIA DCGM Exporter, and the EKS Pod Identity associations they use. Chart values live in |
|
|
The model weights S3 bucket, the |
|
|
The default |
|
|
GPU NodePools, selected by |
|
|
Karpenter path only. EKS Auto Mode covers this functionality natively, so it has |
|
|
Optional add-ons for distributed training, disabled by default. See the following table. |
What a default apply creates
Several components are controlled by input variables. With no -var overrides, terraform apply creates the cluster, the monitoring stack, and the S3 bucket, and does not create any GPU capacity:
| Variable | Default | Effect |
|---|---|---|
|
|
|
No GPU NodePools, so no GPU nodes and no GPU charges. You enable a strategy in Create a GPU NodePool for AI/ML inference. |
|
|
|
Creates the AMP workspace and wires Prometheus remote-write to it. |
|
|
|
Installs the NVIDIA DCGM Exporter. Its Pods stay at zero until GPU nodes join the cluster. |
|
|
|
Skips the EFA Kubernetes device plugin and the MPI Operator. |
|
|
|
Skips the FSx for Lustre file system, CSI driver, and static |
|
|
|
Source CIDR allowed to reach the Grafana ALB. Override this, as described in the following warning. |
Other variables set versions and names, including region (us-east-2), cluster_name (ai-eks-docs), and kubernetes_version. For the full list with descriptions, see variables.tf in your chosen directory. For usage examples beyond this guide, see the READMEterraform/ directory.
Step 1: Deploy Terraform to create the EKS cluster, load balancing, monitoring, and S3 bucket
A single terraform apply creates all of the infrastructure for this section: the EKS cluster, load balancing, the monitoring stack, and the Amazon S3 bucket for model weights.
The repository provides two deployment paths. Choose only one and use it throughout the guide.
-
EKS Auto Mode (
terraform/auto-mode/) — In addition to the core networking, storage, and load balancing add-ons, EKS Auto Mode includes and manages the following capabilities for training and inference workloads: EKS node monitoring agent, automatic node repair, SOCIsnapshotter for fast container pulls, and GPU readiness for the default NodeClass. The NVIDIA device plugin is included in the Bottlerocket accelerated AMI that EKS Auto Mode uses for GPU-enabled nodes. -
Self-managed Karpenter (
terraform/karpenter/) — On an EKS cluster without EKS Auto Mode, the Terraform code installs and configures the components required for training and inference workloads. This includes networking add-ons (VPC CNI, CoreDNS, kube-proxy), Karpenter, the EKS node monitoring agent, the NVIDIA device plugin, and SOCI snapshotter for fast container pulls.
Grafana is publicly accessible over HTTP with default credentials
The Grafana ALB Ingress defaults var.my_cidr to 0.0.0.0/0, which exposes Grafana to the public internet over plain HTTP with default admin credentials. Automated scanners discover public load balancers within minutes. You must restrict access by overriding var.my_cidr with your own IP address:
export MY_CIDR="$(curl -s https://checkip.amazonaws.com)/32" terraform apply -var "my_cidr=${MY_CIDR}"
Treat source-IP allowlisting as a minimum safeguard, not a complete one. Also change the default Grafana admin password after first login. For a stronger posture, change the alb.ingress.kubernetes.io/scheme to internal (reachable only from within your VPC or a connected VPN) and add a TLS certificate.
Change into the directory for your chosen path, initialize Terraform, and apply:
To deploy in a different Region, add -var "region= to the region-code"terraform apply command that follows, where region-code is the AWS Region you want to deploy in.
The Terraform code uses all available Availability Zones in the target Region, excluding use1-az3, usw1-az2, and cac1-az3 because Amazon EKS does not support control plane placement in those zones
Review the Terraform outputs
When the apply completes, Terraform prints the following outputs (values vary based on your configuration):
Apply complete! Resources: 74 added, 0 changed, 0 destroyed. Outputs: cluster_name = "ai-eks-docs" configure_kubectl = "aws eks update-kubeconfig --region us-east-2 --name ai-eks-docs --alias ai-eks-docs" configure_model_bucket = "export MODEL_BUCKET=ai-eks-docs-models-20250612abc1" model_bucket = "ai-eks-docs-models-20250612abc1" node_iam_role_name = "ai-eks-docs-eks-auto-20250612..." region = "us-east-2"
The configure_kubectl output is a ready-to-run command that points kubectl at the cluster. The model_bucket output contains the S3 bucket name for model weights. The node_iam_role_name output shows the IAM role that nodes use.
The resource count differs from the preceding example in Regions with a different number of Availability Zones, because the code creates a subnet, a route table, and their associations per zone.
Configure kubectl
Point kubectl at the new cluster. The configure_kubectl output is a ready-to-run command:
eval "$(terraform output -raw configure_kubectl)"
Verify the cluster
Step 2: Verify the monitoring stack
Terraform already provisioned the full monitoring stack during the terraform apply in Step 1. The stack includes an Amazon Managed Service for Prometheus (AMP) workspace, IAM policies and EKS Pod Identity Associations for Prometheus remote-write and Grafana query access, the kube-prometheus-stack Helm chart (Prometheus, Grafana, kube-state-metrics, node-exporter), and the NVIDIA DCGM Exporter for GPU metrics.
This section covers verification of the deployed monitoring components.
Verify monitoring pods
Wait for all monitoring pods to be ready:
kubectl wait --for=condition=Ready pod --all -n monitoring --timeout=300s kubectl get pods -n monitoring
Expected output:
NAME READY STATUS RESTARTS AGE kube-prometheus-stack-grafana-7c58f54f77-rftrj 3/3 Running 0 5m kube-prometheus-stack-kube-state-metrics-d68dcbc84-5smxq 1/1 Running 0 5m kube-prometheus-stack-operator-5895df479f-ttm47 1/1 Running 0 5m kube-prometheus-stack-prometheus-node-exporter-t9q7s 1/1 Running 0 5m kube-prometheus-stack-prometheus-node-exporter-x6vfb 1/1 Running 0 5m prometheus-kube-prometheus-stack-prometheus-0 2/2 Running 0 5m
Access Grafana
Grafana is exposed through an internet-facing AWS Application Load Balancer (ALB), restricted to the CIDR you set in var.my_cidr. Print the load balancer URL (allow a minute or two for the ALB to provision):
echo "http://$(kubectl get ingress kube-prometheus-stack-grafana -n monitoring -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')"
Open the URL in your browser. Log in with username admin and the password from the following command:
kubectl --namespace monitoring get secrets kube-prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo
Verify the metrics pipeline
To verify the metrics pipeline is working end to end:
-
Navigate to Connections > Data sources and confirm Amazon-Managed-Prometheus is listed as the default datasource.
Validate the AMP datasource in Grafana
-
Navigate to Drilldown > Metrics and search for the
upmetric. You should see results from your cluster’s scrape targets.Validate the
upmetric in Grafana
If up shows results, the pipeline (cluster → Prometheus → AMP → Grafana) is working.
Validate DCGM GPU metrics
The DCGM Exporter DaemonSet runs on GPU nodes and reports GPU utilization, memory, temperature, power draw, NVLink bandwidth, and tensor activity metrics.
Verify the DCGM exporter DaemonSet:
kubectl get daemonset dcgm-exporter -n monitoring
After a GPU node starts running (which you trigger by creating the GPU NodePool in the Create a GPU NodePool for AI/ML inference section), you should see one or more ready Pods. To validate DCGM metrics, navigate to Drilldown > Metrics in Grafana and search for DCGM_. The search returns the GPU metrics the exporter collects, including DCGM_FI_DEV_ECC_SBE_VOL_TOTAL for single-bit ECC errors, DCGM_FI_DEV_ENC_UTIL for encoder utilization, and DCGM_FI_DEV_FB_FREE and DCGM_FI_DEV_FB_USED for free and used framebuffer memory. Any results mean Prometheus is scraping the exporter and writing to Amazon Managed Service for Prometheus.
Validate DCGM metrics in Grafana
To view the pre-provisioned GPU dashboard, navigate to Dashboards > GPU Monitoring > NVIDIA DCGM Exporter Dashboard. The dashboard has panels for GPU Utilization, GPU Avg Temp, GPU Framebuffer Mem Used, and GPU Power Total. Non-zero values in those panels confirm the full path is working, from the exporter on the GPU node through Prometheus to Amazon Managed Service for Prometheus. An idle GPU node reports 0% utilization and 0 MiB framebuffer memory used, but still reports a temperature and power draw.
NVIDIA DCGM Exporter Dashboard in Grafana
Step 3: Verify the model weights S3 bucket
Terraform already created an Amazon S3 bucket for storing model weights, a model-storage-sa ServiceAccount in the default namespace, an IAM policy scoped to the bucket, and an EKS Pod Identity Association that links them. Workload Pods that set serviceAccountName: model-storage-sa can read from and write to the bucket.
Verify the bucket
Retrieve the bucket name from Terraform outputs:
MODEL_BUCKET=$(terraform output -raw model_bucket) echo ${MODEL_BUCKET}
Verify the bucket exists:
aws s3api head-bucket --bucket ${MODEL_BUCKET}
Expected output:
{
"BucketArn": "arn:aws:s3:::ai-eks-docs-models-20250612abc1",
"BucketRegion": "us-east-2",
"AccessPointAlias": false
}
Run a one-off Pod with the AWS CLI image, using the model-storage-sa ServiceAccount, to confirm EKS Pod Identity is wired up and S3 access works:
cat << EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: s3-test labels: guide: ai-eks-docs spec: serviceAccountName: model-storage-sa containers: - name: aws-cli image: public.ecr.aws/aws-cli/aws-cli:2.27.0 command: - sh - -c - | echo "=== Caller Identity ===" aws sts get-caller-identity echo "" echo "=== S3 Write Test ===" echo "pod identity works" | aws s3 cp - s3://${MODEL_BUCKET}/test.txt echo "" echo "=== S3 List Test ===" aws s3 ls s3://${MODEL_BUCKET}/ echo "" echo "=== S3 Delete Test ===" aws s3 rm s3://${MODEL_BUCKET}/test.txt restartPolicy: Never EOF
Wait for the Pod to complete and check the logs:
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/s3-test --timeout=300s kubectl logs s3-test
Expected output:
=== Caller Identity ===
{
"UserId": "AROA...:eks-ai-eks-docs-model-s-...",
"Account": "123456789012",
"Arn": "arn:aws:sts::123456789012:assumed-role/ai-eks-docs-models-.../eks-ai-eks-docs-..."
}
=== S3 Write Test ===
=== S3 List Test ===
2026-07-15 12:00:00 19 test.txt
=== S3 Delete Test ===
delete: s3://ai-eks-docs-models-20250612abc1/test.txtThe caller identity confirms the Pod assumed the model storage role through EKS Pod Identity. The S3 commands confirm read and write access. The write test prints no output of its own, because the AWS CLI suppresses its upload: progress line when the source is standard input; the object listed by the next command is the proof that the write succeeded.
Clean up the test Pod:
kubectl delete pod s3-test
Next steps
With your cluster ready, continue to Create a GPU NodePool for AI/ML inference to create a GPU NodePool that dynamically provisions GPU-enabled nodes for your workloads.
Cleanup
Tip
If you plan to continue with the next sections of this guide, skip the full cleanup. Only run it when you are done.
Delete the test workloads from the GPU NodePool walkthrough first, so no Pod holds a GPU node while the nodes drain:
kubectl delete pod nvidia-smi --ignore-not-found kubectl delete deployment gpu-overflow-test --ignore-not-found
Drain the Karpenter-managed nodes before destroying, so no in-flight node lifecycle blocks the destroy. Delete any PodDisruptionBudgets that would prevent a drain, then delete the NodeClaims:
kubectl delete pdb -A --all --ignore-not-found kubectl delete nodeclaim --all --wait=true --timeout=900s
Then destroy everything Terraform created, including the EKS cluster, the VPC, the monitoring stack, the NodePools and NodeClasses, the S3 model bucket, and any ODCR:
terraform destroy
Important
Pass terraform destroy the same -var overrides you passed to terraform apply, including region and nodepools. Variables are not persisted between commands, so a destroy that omits them falls back to the defaults, targets a different Region than the one you deployed to, and cannot find your resources.
Warning
The model weights S3 bucket is created with force_destroy = true, so terraform destroy deletes the bucket along with any model weights you uploaded to it. Copy anything you want to keep to another location first.
Note
The repository also ships a scripts/cleanup.sh helper that runs the preceding drain and destroy steps and then sweeps any orphaned EBS volumes tagged with the cluster name. Run it from inside the terraform/<mode>/ directory you applied from, and pass --auto-approve to skip the Terraform confirmation prompt.
Confirm no active Capacity Reservation remains for the cluster:
aws ec2 describe-capacity-reservations \ --filters "Name=state,Values=active" "Name=tag:nodepool,Values=reserved-spot-ondemand" \ --query 'CapacityReservations[].CapacityReservationId' \ --output text \ --region [.replaceable]`region-code`
An empty result means no reservation is active and no further charges apply. Run the command against the Region you deployed to, because terraform destroy has already removed the region output by this point.
Important
If you ever interrupted a terraform apply while it was creating the ODCR, Terraform never recorded the reservation in its state and terraform destroy cannot remove it. Drop the tag:nodepool filter to list every active reservation in the Region, then cancel any that belong to this walkthrough with aws ec2 cancel-capacity-reservation --capacity-reservation-id .cr-1234567890abcdef0
Confirm the cluster, its nodes, its VPC, and the model bucket are gone. Run each command against the Region you deployed to, replacing cluster-name with the value of cluster_name you deployed with, which defaults to ai-eks-docs:
aws eks list-clusters --region [.replaceable]`region-code` aws ec2 describe-instances \ --region [.replaceable]`region-code` \ --filters "Name=tag:PartOf,Values=[.replaceable]`cluster-name`" \ "Name=instance-state-name,Values=pending,running,stopping,stopped" \ --query 'Reservations[].Instances[].InstanceId' \ --output text aws ec2 describe-vpcs \ --region [.replaceable]`region-code` \ --filters "Name=tag:PartOf,Values=[.replaceable]`cluster-name`" \ --query 'Vpcs[].VpcId' \ --output text aws s3api list-buckets \ --query "Buckets[?starts_with(Name, '[.replaceable]`cluster-name`-models-')].Name" \ --output text
Each command should return an empty result, other than any unrelated clusters in your account. Terraform tags every resource it creates with PartOf set to the cluster name, so these filters find anything left behind.
Important
If you overrode cluster_name and these commands still use the default, they return an empty result even when resources remain, which looks like a successful cleanup while the resources keep billing. Confirm the name you deployed with before you rely on the output.
