Debugging a HealthOmics workflow run

HealthOmics reports a run's behavior through several sources, each of which answers a different question. For a run that failed, read them in this order instead of starting with the logs:

1. Read the run record for whether the run failed, and why.

2. Read the task records, if a task failed, for which tasks failed and on which instance types. A run can also fail before any task is created, or after every task finished.

3. Read the logs: the task log for what a task reported, the run log for what the run reported outside any task, and the engine log for what the engine reported.

4. Read the run metrics and run manifest log for what the task consumed.

For a run that completed but was slower or more expensive than you expected, start from Compare runs of the same workflow.

The resource and cost guidance in this topic applies to private workflows, for the reason given in Run optimization for a private HealthOmics workflow. For the API operations and response fields that the following steps use, see Get run information.

In the commands in this topic, replace the following values with your own:

run_id

The ID of the run you are debugging.

duration

The window that aws logs tail reads: a relative time such as 2h for a run that finished recently, or a timestamp such as 2026-03-26T22:00:00 for an older one.

run_uuid

The uuid field of that run's record.

task_id

The ID of a failed task in that run.

workflow_id

The ID of the workflow whose runs you want to list.

Diagnose a failed run

To diagnose a failed run

1. Retrieve the run record, and read status, failureReason, statusMessage, and logLocation. Keep the response: later steps use it. logLocation gives the ARNs of the run log (runLogStream) and the engine log (engineLogStream).

   aws omics get-run --id run_id

   Look up the value of failureReason in Run failure reasons. If status isn't FAILED, see Run status values instead.

   Some failures happen before HealthOmics creates any task, or after every task finished, so they leave no task record for the next three steps to read. A role, input, import, or export failure is one; so is an image that HealthOmics rejects before it creates the task (ECR_IMAGE_NOT_FOUND, ECR_PERMISSION_ERROR, ECR_REPOSITORY_NOT_FOUND, or INVALID_ECR_IMAGE_URI). For any of them, skip to the run log later in this procedure. For an image failure, statusMessage names the image URI, so continue from Container images for private workflows and Amazon ECR permissions. If the next step does return a failed task, read that task instead.

2. List the tasks that failed. Each item includes the task's taskId, which the next step uses, plus its name, its creationTime, and, once it reached them, its instance type, startTime, and stopTime.

   aws omics list-run-tasks --id run_id --status FAILED

   If this returns no tasks, the run is one of the cases in step 1: skip the next two steps and read the run log.

3. For each failed task, retrieve its record. It adds failureReason, statusMessage, imageDetails, and logStream to what the previous step returned. logStream is the ARN of the log stream that the next step reads, and the stream's name follows log-stream: in it.

   aws omics get-run-task --id run_id --task-id task_id

4. Read the task log. The events that explain the failure are at the end of the stream.

   aws logs tail /aws/omics/WorkflowLog \
     --log-stream-names run/run_id/task/task_id \
     --since duration

   Set duration to start before the task did: the command reads from then until now, and its default window is the last 10 minutes, so a run that finished earlier returns no events. For the log group and the format of every stream, see Logs in CloudWatch.

   Note

   For why a task might have no log at all, and for ways to make a task more debuggable, see Troubleshooting workflow tasks.

5. Read whichever of the run's other two logs answers the question that brought you here. Both take the step 4 command with a different stream.

   aws logs tail /aws/omics/WorkflowLog \
     --log-stream-names run/run_id \
     --since duration

   The run has no failed task

   Read the run log, run/run_id, as in the preceding command. It reports the run's own progress, including the import and export steps that fail with no task. For what it reports, see Log types for HealthOmics workflows.

   A task failed and its log doesn't explain why

   Read the engine log, run/run_id/engine, which reports what the engine made of the failure, such as the position in the workflow definition. Its ARN is in logLocation.engineLogStream from step 1; if that field isn't in the response, the run has no engine log.

   To read the definition at the position the engine names, export it. The following command returns a download URL for the definition that the run used.

   aws omics get-workflow --id workflow_id --export DEFINITION

Where to continue depends on the failure reason. The following list covers the most common resource and container failures. For what the tasks consumed, which tells you whether the failure was a sizing problem, see Turn findings into cost and sizing changes.

CANNOT_START_CONTAINER_ERROR, CANNOT_START_CONTAINER_SIZE_ERROR, or IMAGE_VERIFICATION_FAILURE

HealthOmics created the task and couldn't start its container, so the task produced no log. Take the image from the task record's imageDetails, then see Container images for private workflows.

OUT_OF_MEMORY_ERROR

Compare the task's memory use against its allocation, as described in How do I right-size the compute and storage for my workflow?, using either its run metrics or its statistics in the run manifest log, which Turn findings into cost and sizing changes locates. Then raise the memory in the workflow definition, and check which instance type the new value selects in Compute and memory requirements for HealthOmics tasks.

TASK_TIMED_OUT or RUN_TIMED_OUT

Confirm whether the task was still doing work when the timeout expired. If it wasn't, treat it as unresponsive rather than slow, and see Guidance for unresponsive runs. If the limit that ended the run came from a run group, see Using HealthOmics run groups.

Inspect resource use with run metrics

Run metrics cover a run that is still in progress, so you can use them before a run finishes. Which series exist for a run depends on its run storage type, its scratch storage mode, and whether its tasks requested accelerators, so check the conditions in Available metrics before you conclude that a metric is missing. If a run has no series at all, check that its role could publish them, as described in Enabling metrics for a run. For the query syntax and the console procedure, see Querying HealthOmics run metrics.

Scope every query to one run with the run resource label. Per-task metrics also carry the task resource label, which joins a series to the task record and its logs. For both labels, see Common attributes.

Note

CloudWatch caps both the number of time series a PromQL query returns and the time range it spans. A run produces at least one series per task for each per-task metric, and more where the metric splits by direction or by GPU. For a run with many tasks, aggregate the series in the query itself, or filter to one task, one direction, or one GPU. Split a long run into consecutive windows. For the current limits, see PromQL limits and restrictions.

Compare runs of the same workflow

When a run is slower or more expensive than an earlier run of the same workflow, compare the two runs task by task. Align the tasks by name rather than by task ID: retrieve the tasks of both runs, group them by name, and compare the duration and the peak values for each name. One read of each run's manifest log gives you every task's timings and, where the entry reports them, its peaks; ListRunTasks gives you the timings for a run that is still going, without the --status FAILED filter that the preceding procedure uses.

Compare two spans per task: startTime to stopTime is the task's duration, and creationTime to startTime is how long it waited to start, which covers its Pending and Starting phases in Task status values. A run can be much slower than another while every task's duration matches, because no task metric covers the wait.

Note

Several tasks in one run can share a name, and a scatter index in a name doesn't necessarily cover the same input data in both runs. Compare the aggregate across the tasks that share a name, such as the longest duration or the highest peak, rather than pairing them index by index.

To establish what is normal for a workflow rather than comparing two runs, aggregate a set of its completed runs. Run Analyzer does this in --batch mode, as described in Impact of file size variance between runs. To collect the runs yourself, filter the response of ListRuns on workflowId. The operation has no workflow filter of its own, so ListRuns pages through every completed run before your filter narrows them.

aws omics list-runs --status COMPLETED \
  --query "items[?workflowId=='workflow_id'].id"

The distribution across a set of runs, not any single run, shows whether a task's allocation fits the variance in your data, and whether a slow-looking run is outside the normal range.

Turn findings into cost and sizing changes

What a task consumed is in the run manifest log, whose stream is manifest/run/run_id/run_uuid. HealthOmics writes it after the run finishes, so set duration to reach the run's stop time rather than a task's. Read it with the same aws logs tail command as the other streams. For the statistics it reports, see Log types for HealthOmics workflows.

A manifest task entry reports what the task reserved, its instance type, and its duration. It reports what the task used alongside those, but not for every task. Where an entry doesn't report usage, or reports less than the task's work can account for, size against the task's run metrics instead, for the reason given in Callouts.

Three HealthOmics behaviors limit what a change saves:

• A task's cost follows the instance type that HealthOmics selects for the resources you request, not the resources the task uses. See Compute and memory requirements for HealthOmics tasks.

• Task charges have a minimum billable duration, so shortening an already short task saves nothing. See Pricing implications of run status.

• Static run storage comes in fixed sizes, so a reduction changes the bill only when it drops to a smaller supported size. See Static run storage.

For the optimization methods that these findings feed into, see Methods to optimize runs.