Configuring custom GRES settings in AWS PCS

Slurm uses gres.conf to manage Generic Resources (GRES), such as GPUs, Multi-Process Service (MPS), and Multi-Instance GPU (MIG) devices, on compute nodes. AWS PCS lets you customize gres.conf through the gresCustomSettings property of slurmConfiguration during compute node group creation or update.

Unlike other custom settings, which are a flat list of parameter and value pairs, gresCustomSettings is a list of records. Each record is a set of gres.conf attributes that describes one GRES device or resource. For example, the record Name=gpu Type=a100 File=/dev/nvidia[0-7] declares eight A100 GPUs.

In Slurm, a gres.conf record applies to every node in the cluster unless it specifies a NodeName= prefix. AWS PCS always adds the NodeName= prefix for the compute node group, so you don't specify it, and every record you supply applies only to that compute node group. AWS PCS doesn't support cluster-wide gres.conf records.

Configuring gres.conf settings

gres.conf custom settings can be configured through the AWS Console, CLI, or SDKs during compute node group creation or modified later through update operations.

AWS Management Console

Navigate to Additional scheduler settings in the create or edit page for a compute node group resource. Enter one gres.conf record per line, in native gres.conf form (for example, Name=gpu Type=a100 File=/dev/nvidia[0-7]).

AWS CLI

For programmatic management of gres.conf settings, use the gresCustomSettings field in create or update compute node group operations. Each element of the list is one gres.conf record.

Example– Declaring GPU devices on a compute node group

aws pcs update-compute-node-group \
    --cluster-identifier my-cluster \
    --compute-node-group-identifier my-cng-1 \
    --slurm-configuration \
    'gresCustomSettings=[{Name=gpu,Type=a100,File=/dev/nvidia[0-7]}]'

Supported gres.conf settings for compute node groups

The following gres.conf attributes are supported in each record at the compute node group level:

• AutoDetect

  Note

  AWS PCS supports AutoDetect on Slurm version 25.11 and later.

• Name

• Type

• Count

• Cores

• Links

• Flags

• File

• MultipleFiles

For the meaning and accepted values of each attribute, see the Slurm gres.conf documentation. AWS PCS validates your records against the same rules that Slurm applies. In particular, each record must declare either a Name or an active AutoDetect value, and a GPU record (Name=gpu) must specify File or MultipleFiles unless an active AutoDetect value discovers the devices.

An active AutoDetect value is any value that turns detection on, such as nvml or full. AutoDetect=off turns detection off, so it is not an active value and can't stand alone in a record.

How AWS PCS applies your gres.conf records

A GRES configuration appears in two places in the Slurm configuration, and AWS PCS writes both of them:

• The gres.conf records that you supply in gresCustomSettings describe the devices that exist on the node.

• The Gres setting in slurm.conf declares what the scheduler can allocate, in the form name[:type]:count. You supply it as a custom Slurm setting for the compute node group, in slurmCustomSettings. For more information, see Custom Slurm settings for AWS PCS compute node groups.

AWS PCS renders your records into gres.conf and reconciles them with the GRES configuration it manages:

• If you declare a GPU record, it replaces the default GPU record that AWS PCS generates for the compute node group. A GPU record is a record that specifies Name=gpu or an AutoDetect attribute. You can declare at most one GPU record for a compute node group.

• Records for other resources, such as mps, shard, or a resource name of your own, are added alongside the AWS PCS defaults.

• AWS PCS derives the GresTypes setting and the gres/* entries of the AccountingStorageTRES setting in slurm.conf from the Gres setting.

On Slurm version 26.05 and later, AWS PCS sets AutoDetect=full by default on GPU compute node groups when the node's socket layout is known, either because you set Sockets in slurmCustomSettings for the compute node group, or because AWS PCS detected the CPU topology automatically. When the socket layout is not available, AWS PCS emits its GPU record without AutoDetect instead.

Constraints between gres.conf and the Gres setting

Slurm needs both sides of the GRES configuration to schedule a resource, and a resource that appears in only one of them doesn't work. An entry in the Gres setting in slurm.conf with no gres.conf record behind it makes the node report fewer resources than are configured, and the node drains. A record with no Gres entry leaves the device present on the node but not schedulable.

AWS PCS validates the two against each other when you create or update a compute node group, and rejects the request in the following cases:

• A slurm.conf Gres entry names a resource that no gres.conf record declares. This includes gpu: a gpu entry requires a GPU record in gresCustomSettings, even when the count is 0.

• A slurm.conf Gres entry declares a higher count of a resource than the gres.conf records declare. A lower count is allowed, down to 0: the gres.conf records describe the devices that are present on the node, while Gres declares how many of them the scheduler can allocate.

• A slurm.conf Gres entry specifies a GPU type that differs from the Type of the GPU record in gres.conf.

The last two checks depend on what your records declare. AWS PCS can't count devices that Slurm discovers when the node boots, so a record that only enables AutoDetect accepts any gpu count and type in the Gres entry. This is how MIG profiles pass validation. AWS PCS also skips the count check for shared GRES (mps and shard), and skips the type check when the GPU record declares no Type.

AWS PCS accepts a record that has no matching Gres entry. The resource exists on the node, but jobs can't request it. Add the Gres entry to make it schedulable.

Important

AWS PCS validates the syntax of your records and their consistency with the Gres setting. It doesn't validate them against the hardware. A record whose File, Cores, or Links layout doesn't match the devices on the booted node makes the node drain. The same applies to MIG: the profiles that you declare in Gres must match the profiles configured on the node. For more information about diagnosing a drained node, see Troubleshoot invalid node registrations (INVALID_REG) in AWS PCS.