Tracking requests with SageMaker HyperPod cluster operations
A cluster operation is a record of a single scaling or on-demand deep health check request on a SageMaker HyperPod cluster. HyperPod creates the operation when it receives your request and tracks it until HyperPod finishes working on it. Each operation has a unique ID, a status, and a record of what the request changed on each instance group. Tracking a request helps you follow it to completion and find the reason if it doesn't finish, without manually correlating cluster events.
With continuous provisioning, requests such as UpdateCluster return as soon
as HyperPod accepts them. HyperPod then brings each instance group to the
requested size in the background, retries failed nodes, and can work on several requests
against the same cluster at the same time. With cluster operations, you can follow each of
those requests:
-
Track a request to completion. See whether HyperPod is still working on your request or has finished.
-
Tell concurrent requests apart. The instance and instance group events that HyperPod records while working on a request carry the operation's ID, so you can list only the events for one request.
-
Find the errors behind a request. Filter a request's events to find the failure reason when, for example, a scale-up doesn't reach its target.
-
Review recent changes. List the requests made on a cluster in the last 14 days and see what each one asked for.
Cluster operations are available for HyperPod clusters orchestrated by Amazon EKS or
Slurm with NodeProvisioningMode set to Continuous. There is no
additional cost.
Topics
Prerequisites
-
Your cluster must have
NodeProvisioningModeset toContinuous.
To use the APIs, your IAM policy must grant the following permissions:
-
sagemaker:ListClusterOperations -
sagemaker:DescribeClusterOperation -
sagemaker:ListClusterEvents, to view an operation's events -
sagemaker:DescribeClusterEvent, to view an operation's events
Requests that create an operation
The following requests create a cluster operation.
| Request | Operation type |
|---|---|
CreateCluster |
Scaling |
UpdateCluster that changes the instance count of one or
more instance groups, or adds or removes an instance group |
Scaling |
BatchAddClusterNodes |
Scaling |
BatchDeleteClusterNodes |
Scaling |
DeleteCluster |
Scaling |
StartClusterHealthCheck (on-demand deep health
check) |
DeepHealthCheck |
Requests made by an autoscaler on your behalf, such as Karpenter calling
BatchAddClusterNodes or BatchDeleteClusterNodes, also create
operations.
The following activity doesn't create an operation, and its events don't carry an operation ID:
-
Automatic node recovery and node replacement, including
BatchReplaceClusterNodes -
Node reboots, including
BatchRebootClusterNodes -
Software updates with
UpdateClusterSoftware -
Deep health checks that run when nodes start (
OnStartDeepHealthChecks)
Cluster operations work together with cluster events. For more information, see SageMaker HyperPod cluster events reference. An operation gives you a way to follow a request through the events it produces, and the activity listed here appears in the same events, without an operation ID.
Operation record
| Field | Description |
|---|---|
OperationId |
Universally unique identifier (UUID) of the operation. |
ClusterArn |
ARN of the cluster. |
Type |
Scaling or DeepHealthCheck. |
Status |
The operation's status. See Operation status. |
CreationTime |
When HyperPod received the request. |
ScalingDetails |
For Scaling operations. One entry for each instance
group the request changed, with that instance group's
Status, its Original counts (before the
request: CurrentCount, TargetCount,
MinCount), and its Desired counts (what the
request asked for: TargetCount,
MinCount). |
DeepHealthCheckDetails |
For DeepHealthCheck operations. One entry for each
instance group checked, with the InstanceIds checked (empty
means every instance in the group) and the DeepHealthChecks
run (InstanceStress, InstanceConnectivity,
MultiNodeTrainingValidation). |
An operation records what you asked for and whether HyperPod is still working on it. It doesn't record the outcome of each step, such as which nodes failed or why. That detail is in the operation's events.
Operation status
| Status | Meaning |
|---|---|
InProgress |
HyperPod is working on the request. |
Done |
HyperPod has finished working on the request. For a
scale-up, the new nodes have run your lifecycle scripts and joined the
orchestrator (Slurm or Kubernetes). ListClusterNodes can
take a few more minutes to show them as Running. For a
scale-down or BatchDeleteClusterNodes, the removed nodes are
gone. A DeleteCluster operation stays InProgress
while the cluster is being deleted. Once the cluster is gone, you can no
longer describe the operation. |
Superseded |
A newer request took over an instance group this operation was changing, so HyperPod now works toward the newer request's target for that instance group. |
Each instance group in ScalingDetails also has its own status. When an
operation changes more than one instance group, a newer request can take over one
instance group while the operation continues on the others. Check the status of each
instance group to see where the operation stands.
Keep the following in mind:
-
There is no
Failedstatus. With continuous provisioning, HyperPod keeps retrying until it reaches the target. An operation that can't make progress, for example because a lifecycle script fails on every new node, staysInProgressuntil it expires after 14 days. To find out why an operation isn't finishing, list itsErrorevents. For more information, see Find the errors behind an operation. -
A failed cluster creation shows
Done. IfCreateClusterfails, its operation readsDone. Check the cluster's status withDescribeCluster. -
A
DeepHealthCheckoperation doesn't record the check result. When the operation reachesDone, the check has finished. To see whether it passed, read the operation's events. -
Events can arrive after an operation ends. Work that was already under way when a newer request took over, such as an instance that was launching, can continue, and its events still carry the superseded operation's ID.
-
Work after an operation is
Doneisn't part of the operation. If HyperPod later retries or replaces a node that the operation brought up, those events have no operation ID.
Viewing operations in the console
-
Open the SageMaker AI console
. -
In the left navigation pane, choose HyperPod clusters.
-
Choose your cluster name.
-
Choose the Events tab.
The Events tab shows the operations on your cluster alongside its
events. To see only the events for one operation, filter the Events
tab by operation ID. In the Instance groups table, the
Operation ID column shows the operation currently working on each
instance group. Use this ID to filter the Events tab to that
operation's events. When an operation that is in progress has Error events,
the console shows an alert with a link to that operation's error events.
Listing operations using the AWS CLI
Use the list-cluster-operations command to list operations newest
first:
aws sagemaker list-cluster-operations \ --cluster-name my-cluster \ --sort-by CreationTime \ --sort-order Descending
Narrow results with filters:
-
--status(InProgress,Done, orSuperseded) -
--type(ScalingorDeepHealthCheck) -
--creation-time-afterand--creation-time-before(a time window)
For example, to list scaling operations still in progress:
aws sagemaker list-cluster-operations \ --cluster-name my-cluster \ --status InProgress \ --type Scaling
The following is example output.
{ "ClusterOperationSummaries": [ { "OperationId": "4c078808-1f2e-4a6b-9c3d-8e5f7a2b1c0d", "ClusterArn": "arn:aws:sagemaker:us-west-2:111122223333:cluster/abcd1234efgh", "Type": "Scaling", "Status": "InProgress", "CreationTime": "2026-11-20T13:45:02.118000-08:00" } ] }
To find the operation for a request you just made, list the operations on the cluster
sorted by CreationTime and look for the operation created at the time of
your request.
Describing an operation
Use the describe-cluster-operation command:
aws sagemaker describe-cluster-operation \ --cluster-name my-cluster \ --operation-id 4c078808-1f2e-4a6b-9c3d-8e5f7a2b1c0d
The following is example output for a scale-up of the gpu-workers
instance group from 4 to 8 nodes.
{ "OperationId": "4c078808-1f2e-4a6b-9c3d-8e5f7a2b1c0d", "ClusterArn": "arn:aws:sagemaker:us-west-2:111122223333:cluster/abcd1234efgh", "Type": "Scaling", "Status": "InProgress", "CreationTime": "2026-11-20T13:45:02.118000-08:00", "ScalingDetails": [ { "InstanceGroupName": "gpu-workers", "Status": "InProgress", "Original": { "CurrentCount": 4, "TargetCount": 4, "MinCount": 0 }, "Desired": { "TargetCount": 8, "MinCount": 0 } } ] }
To wait for a request to finish, call describe-cluster-operation
periodically until Status is Done or
Superseded. Because there is no Failed status, set a timeout
and check the operation's Error events if the operation stays
InProgress longer than expected. For DeleteCluster, wait until
DescribeCluster returns ResourceNotFound. Once the cluster is
gone, describe-cluster-operation returns ResourceNotFound
too.
Viewing an operation's events
Use the --operation-id filter on list-cluster-events:
aws sagemaker list-cluster-events \ --cluster-name my-cluster \ --operation-id 4c078808-1f2e-4a6b-9c3d-8e5f7a2b1c0d
You can combine --operation-id with the other
list-cluster-events filters. For example, add --node-id to
trace one node's events within an operation. To go the other way, from an event to the
request that caused it, read the event's OperationId field. Events that
don't belong to any operation, such as cluster-level events and node replacement events,
have no OperationId.
Find the errors behind an operation
If an operation stays InProgress or an instance group doesn't reach its
target:
-
List the operation's events with
--operation-id. -
Look for events with
EventLevelset toError. -
Call
describe-cluster-eventon an error event to read itsFailureMessage, for example an insufficient-capacity error or a lifecycle script failure with a link to its CloudWatch log group.
In the console, choose the link in the error alert, or filter the
Events tab by the operation's ID and look for Error
events.
Retention
Operations are available for 14 days, the same as cluster events. Operations aren't an audit log. To keep a permanent record of the API calls made on your cluster, use CloudTrail. In rare cases, a request might not have an operation, for example if HyperPod couldn't record it. The request itself isn't affected.
Related topics
-
ListClusterOperations and DescribeClusterOperation in the Amazon SageMaker AI API Reference