Troubleshooting with pcluster-diag

pcluster-diag is a diagnostics tool that verifies a set of conditions that a healthy AWS ParallelCluster node is expected to satisfy.

Use pcluster-diag as a first step when a cluster misbehaves at runtime. The tool emits a JSON diagnostics report to help you understand the problems that are affecting your cluster. If you can't resolve a problem, attach the report to an AWS Support case.

The tool is included in every AWS ParallelCluster AMI starting with version 3.16.0, both the official AMIs and the custom AMIs that you build with pcluster build-image.

You can run pcluster-diag on any cluster node. If you don't know where the problem is, run it on the head node.

Key characteristics

• Context aware – at startup it reads the node type and the deployed cluster configuration, then runs only the checks that apply. Checks for features that your cluster doesn't use are reported as skipped.

• Read-only by default – it never changes the configuration of your cluster. A check that isn't read-only requires your explicit approval before it runs, and records it as skipped if you decline.

• Complete in a single run – a check that fails never stops the others. Every applicable check runs on every invocation, so one run gives you the complete picture of the node, according to the available checks.

Available checks

To see which checks the tool will execute, use the describe-checks subcommand, which returns a JSON array of every registered check, each with its id and description.

$ pcluster-diag describe-checks
[
  {
    "check_id": "...",
    "check_description": "..."
  },
  ...
]

Run pcluster-diag

Connect to the node that you want to diagnose, and then execute the run subcommand as root.

$ sudo pcluster-diag run

pcluster-diag writes two separate streams:

• A progress log to standard error.

• The JSON report to standard output. The same report is also saved to a timestamped file under ./pcluster-diag-output/ in the current directory.

The following options are available.

--output-file path

The file that the JSON report is written to. The default is a timestamped file under ./pcluster-diag-output/, for example ./pcluster-diag-output/pcluster-diag-report-2026-08-04T10-12-00.json.

-y, --yes

Approve every check that requires confirmation, without prompting. Use this when you run pcluster-diag from a script.

--version

Print the pcluster-diag version.

--help

Print the usage information. pcluster-diag run --help prints the options of the run subcommand.

Interpret the report

pcluster-diag emits a JSON report that accounts for every check it ran, whether the check passed, warned, failed, or was skipped. context describes the node that was diagnosed. The general structure of the report is as follows:

{
  "context": { ... },
  "results": [
    {
      "check_id": "...",
      "check_description": "...",
      "status": "...",
      "errors": [ { "code": "...", "message": "..." } ],
      "warnings": [ { "code": "...", "message": "..." } ],
      "infos": [ { "code": "...", "message": "..." } ]
    }
  ]
}

Check statuses

The status of each check tells you what to do with it.

Status	Meaning	What to do
PASSED	This check found no problem.	-
WARNING	This check found something that might cause a problem on the node.	Review the warnings list and address anything that might be important.
FAILURE	This check found a problem on the node.	Review the errors list and address all the problems.
CHECK_ERROR	The check couldn't complete, so status couldn't be confirmed.	Treat it as inconclusive, not as a problem. Report the error to AWS Support because it might be the signal of an unhandled error.
SKIPPED_NOT_APPLICABLE	The check doesn't apply to this node type or to your cluster configuration.	Nothing. This is the expected status for features that your cluster doesn't use.
SKIPPED_BY_USER	The check required your confirmation and you declined it.	Rerun pcluster-diag, approving the check when prompted.

If every check reports PASSED, pcluster-diag found no problems. However, the coverage of pcluster-diag is not comprehensive and will grow with every release. A healthy diagnosis does not guarantee that the cluster is healthy.

Check findings

A check result can carry three kinds of findings, each with a code and a message.

• errors, coded En, are the reasons a check failed. The reserved code E0 is used only for internal errors that prevented the correct execution of the check.

• warnings, coded Wn, are non-fatal observations.

• infos, coded In, are contextual notes.

Get the latest updates from GitHub

New checks and improved diagnostic messages land in the aws-parallelcluster-cookbook repository on the GitHub website. To use the latest checks without waiting for the next release, you can refresh the copy of the tool on a running node in place.

Run the following on the node that you want to update. It replaces the tool source under /opt/parallelcluster/sources/pcluster-diag with the version on the develop branch.

$ curl -fL https://github.com/aws/aws-parallelcluster-cookbook/archive/refs/heads/develop.tar.gz \
  | sudo tar -xz --strip-components=5 -C /opt/parallelcluster/sources/pcluster-diag \
    --wildcards '*/cookbooks/aws-parallelcluster-platform/files/pcluster-diag/*'

Confirm which version you now have:

$ sudo pcluster-diag --version

To use a version other than the tip of develop, replace develop in the URL with the branch or the tag you want, for example a release-* branch.

Keep the following in mind:

• The refresh applies only to the node that you run it on. Repeat it on every node that you want to diagnose with the newer checks.

• The tool source is baked into the AMI, so a node that gets replaced comes back with the version from the AMI.