Hung job detection
A training job can hang without exiting, for example because of an NVIDIA Collective Communications Library (NCCL) timeout, a data pipeline stall, or a process deadlock. The job keeps its GPUs but makes no progress. Hung job detection finds these jobs and writes an event to Amazon CloudWatch, and can optionally cancel the job.
Hung job detection runs as an agent on each node of a SageMaker HyperPod Slurm cluster. It's
off by default and you turn it on for each job with the HYPERPOD_HANG_RULES
environment variable. It needs no changes to your training scripts or containers, and
works with PyTorch, TensorFlow, JAX, and custom scripts.
To use hung job detection on an existing cluster, you must update the cluster software to the latest AMI. Otherwise, hung job detection isn't available on that cluster. For more information, see Update the SageMaker HyperPod platform software of a cluster.
Turning on hung job detection for a job
The rules file must be readable from every compute node in the job. For multi-node jobs, use shared storage such as Amazon FSx for Lustre.
To turn on hung job detection
-
Create a rules file that turns on hung job detection.
echo '{"enabled": true}' > /fsx/my-project/rules.json -
When you submit the job, set the
HYPERPOD_HANG_RULESenvironment variable to the path of the rules file.#SBATCH --export=ALL,HYPERPOD_HANG_RULES=/fsx/my-project/rules.json
With this file, the agent uses default detection, which watches the job's output and GPU activity, and only writes an event when it finds a hang. For details, see How hung job detection works. To add your own rules or cancel hung jobs, see Configuring custom detection rules.
How hung job detection works
A job that doesn't set HYPERPOD_HANG_RULES isn't monitored.
If the rules file has no enabled log_pattern rules, the agent uses default detection. It declares a job hung when either of the following is
true:
-
Job output (stdout) has stopped for an extended period while the GPUs are idle but still hold memory. This catches rendezvous hangs, where one node fails and the others wait at a barrier.
-
Job output has stopped for a longer period, regardless of GPU state.
If the rules file has log_pattern rules, the agent watches your log
file for those patterns instead. A job is hung when a pattern stops matching for its
timeout, or immediately when a fault_on_match pattern appears.
Custom rules replace default detection for that job. If your patterns keep matching, the agent doesn't declare a hang even when the GPUs are idle.
Hung job detection is independent of auto-resume, which restarts jobs that exit after a hardware failure. For more information about auto-resume, see Automatic node recovery and auto-resume.
Viewing detection events
HyperPod writes detection events from the controller node to your cluster's CloudWatch log group:
-
Log group:
/aws/sagemaker/Clusters/cluster-name/cluster-id -
Log stream:
SageMakerHangJobDetectionEvents/controller-instance-group-name/controller-instance-id
Events for every job are written to this stream, whichever node the job ran on. Each
event's node field shows the Slurm node where the hang was detected. To
find the events for a job, filter the log group by its job ID:
aws logs filter-log-events \ --log-group-name /aws/sagemaker/Clusters/cluster-name/cluster-id\ --log-stream-name-prefix SageMakerHangJobDetectionEvents/ \ --filter-pattern '{ $.job_id = "12345" }'
The following is an example HANG_DETECTED event.
{ "event_type": "HANG_DETECTED", "job_id": "12345", "step_id": "0", "node": "ip-172-22-230-3", "evidence": "stdout stale for 18 min and GPU idle with memory allocated", "layer": "signal_evaluator", "detected_at": "2026-04-22T00:58:47Z", "job_action": "notify", "config_source": "per_job_env_var" }
The layer field shows what fired: signal_evaluator for default detection or pattern_evaluator for your rules.
job_action is notify or cancel.
The log stream can also contain the following events:
-
HANG_ACTION_TAKEN: the agent cancelled the job. Theactionfield shows what Slurm did. By default, Slurm requeues jobs on SageMaker HyperPod Slurm clusters, so a cancel usually reportsrequeue. -
CONFIG_ERROR: if the rules file is missing or invalid, the agent doesn't monitor the job. If a pattern doesn't compile, the agent skips that rule. The event includes the file path and the error. This error doesn't appear insbatch,squeue,sacct, or the job's output, so test a new rules file with a short job and check the log stream before you use it for production jobs.
Configuring custom detection rules
We recommend that you add at least one log_pattern rule that tracks your
training progress, such as steps or epochs. Default detection can't catch a job
that keeps its GPUs busy without making progress.
A rules file is a JSON file. The following example watches for epoch completion and cancels the job if no epoch completes within 45 minutes, or as soon as an NCCL error appears.
{ "enabled": true, "rules": [ { "name": "epoch_progress", "type": "log_pattern", "enabled": true, "log_pattern": "Epoch \\d+/\\d+ complete", "timeout_minutes": 45 }, { "name": "nccl_error", "type": "log_pattern", "enabled": true, "log_pattern": "NCCL error|NCCL WARN", "fault_on_match": true } ], "action_policy": { "action": "cancel" } }
The following table describes the fields in a rules file.
| Field | Required | Description |
|---|---|---|
enabled |
Yes | Set to true to turn on detection for jobs that use
this file. If it's false or missing, those jobs aren't
monitored. |
monitoring.log_file |
No | Log file to watch. Defaults to the job's Slurm output file (the
--output file, or
slurm-). |
rules |
No | Up to 20 rules. Enabled log_pattern rules replace default detection. |
action_policy.action |
No | notify (default) writes an event only.
cancel also cancels the job. Slurm requeues
cancelled jobs by default; to stop the job instead, submit it with
#SBATCH --no-requeue. |
The following table describes the fields in each rule.
| Field | Required | Description |
|---|---|---|
name |
Yes | Rule name, shown in events. |
type |
Yes | log_pattern. This is the only rule type the agent
evaluates. |
enabled |
No | Defaults to false. Set to true to use
the rule. |
log_pattern |
Yes | RE2 regular expression, up to 256 characters. |
timeout_minutes |
Yes, unless fault_on_match is
true |
How long the pattern can go unmatched before the job is hung. |
fault_on_match |
No | If true, a match means the job is hung. Use for
error patterns. |
stop_pattern |
No | A pattern that turns off the rule when it matches, for example
Training complete. |
start_timeout_minutes |
No | Maximum time from job start to the first match. Catches jobs that hang during startup. |
metric_evaluation_data_points |
No | Number of consecutive 10-second checks that must fail before
the job is hung. Default is 1. Raise it to reduce
false positives. |
To apply one rules file to many jobs, set HYPERPOD_HANG_RULES in
your batch script, your shell profile, or a site-wide profile script. If you aren't sure how sbatch propagates your environment, set the variable on the #SBATCH --export line.
To make sure a change to a rules file takes effect, cancel the job and submit it again.
Turning off hung job detection
A job that doesn't set HYPERPOD_HANG_RULES isn't monitored. To turn
off detection in other cases, do one of the following:
-
For one job, add
--no-hang-detectiontosrun, or tosbatchto cover all steps in the job.srun --no-hang-detection python train.py -
For all jobs that use a rules file, set
"enabled": falsein the file.
Securing rules files
The agent applies the rules file that a job points to. It doesn't check who owns the file or who can change it. Anyone who can write to the file, or replace it in its directory, can change detection for every job that uses it, including making the agent cancel those jobs.
Keep each rules file, and the directory that contains it, writable only by its owner. For a rules file that a team shares, let the team read it and let only the person who maintains it write to it.
# Rules file for your own jobs chmod 600 /fsx/my-project/rules.json # Shared rules file: others can read, only the owner can write chmod 644 /fsx/team/rules.json ls -l /fsx/team/rules.json
Limitations
-
Default detection doesn't catch a job that keeps its GPUs busy without making progress, such as a job stuck in a loop. To catch these, we recommend that you always add a
log_patternrule that tracks steps or epochs. -
Default detection waits for a sustained period of inactivity, so detection isn't immediate. Shorter custom timeouts detect faster but can flag normal pauses such as checkpoint saves.
-
Patterns use RE2 syntax. A pattern that uses backreferences or lookaround doesn't compile, and the agent skips that rule.
-
A rules file can have up to 20 rules and can be up to 1 MB. A file over either limit is rejected and the job isn't monitored.