Configuring Multi-Instance GPU (MIG) in AWS PCS

NVIDIA Multi-Instance GPU (MIG) partitions a supported GPU into isolated GPU instances, each with its own memory and compute resources. Slurm schedules each MIG profile as its own Generic Resource (GRES) type, so multiple jobs can share one physical GPU with hardware isolation.

MIG devices can't be described statically in gres.conf: their device files and UUIDs only exist after the MIG instances are created on the booted node, so there is no File or Type value to configure ahead of time. On AWS PCS you configure MIG with a gres.conf record that only enables AutoDetect, together with a Gres setting that declares the profiles. slurmd then discovers the MIG devices when the node boots. AWS PCS supports AutoDetect, and therefore MIG, on Slurm version 25.11 and later.

Prerequisites

• A compute node group that uses a MIG-capable instance type.

• Slurm version 25.11 or later.

• Configured CPU topology. GPU autodetection requires the node's socket layout. On Slurm 25.11, set Sockets for the compute node group; on Slurm 26.05 and later, AWS PCS configures it automatically. For more information, see Configuring hardware topology in AWS PCS.

Step 1: Configure MIG on the instances

The MIG instances must exist before slurmd starts and registers the node. If MIG mode is enabled but no MIG instances are created, slurmd detects the parent GPUs, finds no usable MIG device, and the node registers with no schedulable GPU. Create the MIG instances in the node's boot process, for example in a custom AMI or a launch template user data script that runs before slurmd starts. For more information, see NVIDIA Multi-Instance GPU User Guide.

1. Enable MIG mode on the GPUs:

   nvidia-smi -mig 1

   MIG mode is GPU state, not disk state, and it resets between EC2 instance allocations, so it must be enabled on every launch; it can't be baked into an AMI as already enabled. It also can't take effect while processes are using the GPUs. Enable it early in the boot sequence, before any service that uses the GPUs starts (for example, from a systemd unit in a custom AMI or a cloud-init boothook in the launch template user data; see Example: Run early boot operations with a cloud-init boothook). At that point no process holds the GPUs, so the mode change takes effect without a reboot. Verify that MIG mode is Enabled (not Pending) on every GPU with nvidia-smi -q.

   Important

   If your platform does require a reboot to change MIG mode, reboot from an early boot stage such as a cloud-init boothook, before the AWS PCS bootstrap starts. Don't reboot from a node lifecycle action script: a reboot there interrupts the AWS PCS bootstrap sequence. For how to create the boot hook, see Example: Run early boot operations with a cloud-init boothook.

2. Create the GPU instances and compute instances for the profiles you want. For example, to partition each of 8 GPUs into one 3g.20gb, one 2g.10gb, and two 1g.5gb instances:

   for i in 0 1 2 3 4 5 6 7; do
       nvidia-smi mig -i $i -cgi 9,14,19,19 -C
   done

Step 2: Configure the compute node group

Declare both halves of the GRES configuration on the compute node group:

• In gresCustomSettings, a GPU record that only enables AutoDetect. Don't set Name, Type, or File. The MIG devices are discovered on the node.

• In slurmCustomSettings, a Gres setting that declares the MIG profiles and counts that jobs can request. The profile names and counts must match the MIG instances created on the node.

Example– Declaring MIG profiles on a compute node group (8 GPUs partitioned as shown in Step 1)

aws pcs update-compute-node-group \
    --cluster-identifier my-cluster \
    --compute-node-group-identifier my-cng-1 \
    --slurm-configuration \
    'gresCustomSettings=[{AutoDetect=nvml}],slurmCustomSettings=[{parameterName=Gres,parameterValue=gpu:3g.20gb:8\,gpu:2g.10gb:8\,gpu:1g.5gb:16}]'

For the constraints between the two halves, see Constraints between gres.conf and the Gres setting. Because AWS PCS can't count devices that are discovered at boot, it accepts any profile names and counts in the Gres setting for a record that only enables AutoDetect. Consistency with the actual MIG instances is only verified when the node registers.

When a node boots and registers, it advertises the discovered profiles with their socket affinity, for example Gres=gpu:3g.20gb:8(S:0),gpu:1g.5gb:28(S:1). A job that requests a profile, such as --gres=gpu:3g.20gb:1, is placed on a single MIG instance and gets its UUID in CUDA_VISIBLE_DEVICES (for example, MIG-e7aa6185-06d5-53e9-a4d9-00f7670f741e), so CUDA restricts the job to that partition.

Note

The controller logs warnings such as Ignoring file-less GPU gpu:3g.20gb from final GRES list for MIG records. This is expected: the authoritative GRES comes from node registration.

Limitations

Because a MIG configuration names no devices, the controller only knows a node's real GRES after the node boots and registers. This causes the following limitations. They apply to any GPU record that only enables AutoDetect, and therefore to every MIG compute node group.

Note

As a best practice, use a static compute node group for MIG: set minInstanceCount equal to maxInstanceCount in the scaling configuration. The nodes then stay running and registered instead of powering down between jobs, which mitigates both limitations.

Configuration updates don't reach powered-down nodes

When you change the MIG configuration of a compute node group (the profiles in the Gres setting, or the MIG partitioning in the boot process), the change doesn't reach nodes that previously registered the old configuration and are powered down at that moment. Those nodes keep advertising the profiles they last registered until they next boot and register, which only happens when a job is allocated to them. The consequences are:

• Jobs that request the new profiles are rejected at submission with Requested node configuration is not available, because no node advertises them yet.

• Jobs that request the old profiles are still accepted, even though the configuration no longer declares them.

• If you reduce a profile count, a job sized for the old count can be admitted and run on a node that has fewer MIG instances than the job requested.

After you change the MIG configuration, boot every node that registered the previous configuration so it re-registers and advertises the new profiles. To boot a powered-down node without submitting a job to it, run scontrol from a login node:

scontrol update NodeName=my-cng-1-[1-8] State=POWER_UP

Jobs allocated to powered-down nodes can start without a GPU

A job allocated to a node while the node is powered down is granted a profile count, but no specific device, because the devices aren't known yet. When the node then boots and the job starts, the job has no MIG device assigned: it runs with an empty CUDA_VISIBLE_DEVICES, and with ConstrainDevices set in the cgroup configuration it is denied access to every GPU. Slurm reports the job as RUNNING though. The MIG instance the job was counted against isn't assigned to any other job either, so it stays unusable until the job ends.

To detect this condition, verify at the start of the job that CUDA_VISIBLE_DEVICES is not empty, and exit or requeue the job when it is:

if [ -z "$CUDA_VISIBLE_DEVICES" ]; then
    scontrol requeue "$SLURM_JOB_ID"
    exit 1
fi