# Upgrade the task governance add-on Use this section to upgrade the HyperPod task governance Amazon EKS add-on between versions. Each subsection provides version-specific procedures for upgrading your add-on while preserving your existing configuration. **Topics** + [Upgrade from v1.5 to v1.6](#hp-eks-task-governance-upgrade-v15-to-v16) + [Upgrade from v1.3.x to v1.5 or later](#hp-eks-task-governance-upgrade-v13-to-v15) ## Upgrade from v1.5 to v1.6 Version v1.6.0-eksbuild.1 packages Kueue v0.19.2. You can upgrade directly from v1.5 with `aws eks update-addon`. This upgrade does not migrate custom resource definition (CRD) storage versions, so you do not need to back up or re-create any Kueue objects. To upgrade the add-on to v1.6 through the Amazon EKS add-on interface, run the following command. Replace {{region}} with your Region and {{cluster-name}} with your Amazon EKS cluster name. ``` aws eks update-addon --region {{region}} --cluster-name {{cluster-name}} \ --addon-name amazon-sagemaker-hyperpod-taskgovernance \ --addon-version v1.6.0-eksbuild.1 --resolve-conflicts OVERWRITE ``` Wait until the status is `ACTIVE`: ``` aws eks describe-addon --region {{region}} --cluster-name {{cluster-name}} \ --addon-name amazon-sagemaker-hyperpod-taskgovernance \ --query 'addon.status' --output text ``` **Topology-aware scheduling slice size validation** Beginning with v1.6 (Kueue v0.19.2), task governance validates the topology-aware scheduling slice size (the number of pods grouped together for placement) when you submit a job. If you submit a job that sets the `kueue.x-k8s.io/podset-slice-size` annotation to a value less than `1`, task governance rejects the job with an error like the following: ``` admission webhook "vjob.kb.io" denied the request: spec.template.metadata.annotations[kueue.x-k8s.io/podset-slice-size]: Invalid value: "0": must be greater than or equal to 1 ``` Earlier versions did not validate this value. Before you upgrade, update any job templates that use topology-aware scheduling slices to set `kueue.x-k8s.io/podset-slice-size` to `1` or greater. ## Upgrade from v1.3.x to v1.5 or later The recommended way to upgrade from v1.3.x is the upgrade option in the SageMaker AI HyperPod console, which migrates the Kueue CRDs automatically. Use the manual procedure in this section only if you cannot use the console. This procedure supports v1.5 and later as the target version. A direct `aws eks update-addon` from v1.3.x to v1.5 or later fails because v1.3.x stores some Kueue custom resource definitions (CRDs) under the `v1alpha1` API version, which v1.5 removes: ``` CustomResourceDefinition.apiextensions.k8s.io "cohorts.kueue.x-k8s.io" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": missing from spec.versions ``` This procedure backs up your Kueue objects and clears the old stored version. It then upgrades the add-on and restores your objects under the new schema. **Data impact and timing** This procedure deletes and re-creates your Kueue custom resource objects (ClusterQueues, LocalQueues, ResourceFlavors, Topologies, and related objects). It backs them up first and restores them, so no configuration is lost. It does not delete any namespace, does not delete any CRD, and does not change any SageMaker AI ComputeQuota or ClusterSchedulerConfig record. Run this when no new workloads need to be submitted. Running pods are generally not interrupted, but we recommend not relying on active workloads during the migration. New workloads cannot be scheduled until the procedure completes. Run against one cluster at a time. ### Prerequisites Before you begin, make sure you have the following: + `kubectl` configured for the target Amazon EKS cluster with cluster-administrator access + The AWS CLI configured for the cluster's account and Region + `jq` installed + The add-on is currently at v1.3.x with status `ACTIVE` or `DEGRADED` Throughout, replace {{region}} with your Region and {{cluster-name}} with your Amazon EKS cluster name. To upgrade the add-on from v1.3.x to v1.5, complete the following steps: 1. **Confirm the current add-on version and set a working directory.** ``` aws eks describe-addon --region {{region}} --cluster-name {{cluster-name}} \ --addon-name amazon-sagemaker-hyperpod-taskgovernance \ --query 'addon.addonVersion' --output text ``` Confirm the output begins with `v1.3.`. Then set `BACKUP_DIR` to an absolute path in a writable directory and create it. Later steps read from and write to this variable, so run every step in the same shell session. ``` export BACKUP_DIR={{/absolute/path/to/backup-dir}} mkdir -p "$BACKUP_DIR" ``` 1. **Back up every Kueue custom resource to local files.** ``` for crd in admissionchecks clusterqueues cohorts localqueues multikueueclusters \ multikueueconfigs provisioningrequestconfigs resourceflavors topologies \ workloadpriorityclasses workloads; do kubectl get "${crd}.kueue.x-k8s.io" --all-namespaces -o json \ > "$BACKUP_DIR/${crd}.json" 2>/dev/null echo "${crd}: $(jq '.items | length' "$BACKUP_DIR/${crd}.json" 2>/dev/null || echo 0)" done ``` **Verify the backup before continuing** Confirm that the backup directory contains a JSON file for each custom resource in the preceding command, and that the object counts in the command output match what your cluster had. Do not proceed if any file is missing or empty. 1. **Delete the backed-up objects and clear the old stored version from each CRD.** This removes the `v1alpha1` (or `v1beta1`) entry from `status.storedVersions` so the v1.5 or later CRDs can install. The objects are safe in your backup and are restored in a later step. ``` for crd in admissionchecks clusterqueues cohorts localqueues multikueueclusters \ multikueueconfigs provisioningrequestconfigs resourceflavors topologies \ workloadpriorityclasses workloads; do kubectl get crd "${crd}.kueue.x-k8s.io" >/dev/null 2>&1 || continue kubectl delete "${crd}.kueue.x-k8s.io" --all --all-namespaces \ --ignore-not-found=true --wait=false --request-timeout=30s kubectl patch crd "${crd}.kueue.x-k8s.io" --subresource=status --type=merge \ --request-timeout=30s -p '{"status":{"storedVersions":["v1beta2"]}}' done ``` **About --all-namespaces and --wait=false** `--all-namespaces` here selects custom resources across all namespaces to delete; it does not delete any namespace. `--wait=false` avoids blocking on finalizers. The add-on update in the next step resolves them. 1. **Update the add-on to your target version (v1.5.0-eksbuild.1 or later).** Replace {{target-version}} with the target add-on version, for example `v1.5.0-eksbuild.1` or `v1.6.0-eksbuild.1`. ``` aws eks update-addon --region {{region}} --cluster-name {{cluster-name}} \ --addon-name amazon-sagemaker-hyperpod-taskgovernance \ --addon-version {{target-version}} --resolve-conflicts OVERWRITE ``` Wait until the status is `ACTIVE`: ``` aws eks describe-addon --region {{region}} --cluster-name {{cluster-name}} \ --addon-name amazon-sagemaker-hyperpod-taskgovernance \ --query 'addon.status' --output text ``` 1. **Wait for the new install to settle before restoring.** Do not restore immediately after the add-on reports `ACTIVE`. Wait for the controller, its webhook, and the post-install jobs to be ready, or the restore in the next step can hang. ``` kubectl rollout status deploy/kueue-controller-manager -n kueue-system --timeout=300s ``` ``` until [ -n "$(kubectl get endpoints -n kueue-system kueue-webhook-service \ -o jsonpath='{.subsets[*].addresses[*].ip}' 2>/dev/null)" ]; do echo "waiting for kueue webhook endpoint..."; sleep 5 done ``` ``` kubectl wait --for=condition=complete job -l app.kubernetes.io/name=kueue \ -n kueue-system --timeout=180s || true ``` 1. **Restore your objects under the new schema.** This transforms each backed-up object to the v1.5 or later (`v1beta2`) schema and re-applies it. ``` transform() { jq ' .apiVersion = "kueue.x-k8s.io/v1beta2" | del(.status) | del(.metadata.resourceVersion, .metadata.uid, .metadata.creationTimestamp, .metadata.generation, .metadata.managedFields, .metadata.selfLink) | del(.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration") | if .kind == "Cohort" and (.spec.parent != null) then .spec.parentName = (.spec.parentName // .spec.parent) | del(.spec.parent) else . end | if .kind == "ClusterQueue" and (.spec.cohort != null) then .spec.cohortName = (.spec.cohortName // .spec.cohort) | del(.spec.cohort) else . end | if .kind == "ClusterQueue" then del(.spec.admissionChecks) else . end | if .kind == "AdmissionCheck" then del(.spec.retryDelayMinutes) else . end ' } for crd in resourceflavors topologies workloadpriorityclasses admissionchecks cohorts \ provisioningrequestconfigs multikueueclusters multikueueconfigs \ clusterqueues localqueues workloads; do f="$BACKUP_DIR/${crd}.json" [ -s "$f" ] || continue count=$(jq '.items | length' "$f") for (( i=0; i/dev/null 2>&1; then echo "applied $name" else echo "check $name (may already be recreated by the add-on)" fi done done ``` 1. **Verify the result.** ``` aws eks describe-addon --region {{region}} --cluster-name {{cluster-name}} \ --addon-name amazon-sagemaker-hyperpod-taskgovernance \ --query 'addon.{version:addonVersion,status:status}' ``` An example output is as follows (your `version` matches the {{target-version}} you set in step 4). ``` { "version": "v1.5.0-eksbuild.1", "status": "ACTIVE" } ``` Confirm your objects are present and no CRD still lists `v1alpha1`: ``` kubectl get clusterqueues kubectl get localqueues --all-namespaces kubectl get crd clusterqueues.kueue.x-k8s.io -o jsonpath='{.status.storedVersions}' ``` The `storedVersions` output must contain only `v1beta2` (or `v1beta1` and `v1beta2`), never `v1alpha1`. Compare the restored objects against the files in `$BACKUP_DIR` to confirm that your configuration values are unchanged.