Configuring hardware topology in AWS PCS

Slurm makes better placement decisions when it knows the internal hardware layout of a compute node: how cores group into sockets and Non-Uniform Memory Access (NUMA) domains, how L3 caches map to cores, and which cores each GPU is attached to. This topic describes how Slurm models that layout, and how AWS PCS configures it on each supported Slurm version.

CPU topology

The cores of a compute node are not interchangeable. They are grouped into physical packages (sockets) and NUMA domains, and each group shares resources such as memory controllers and L3 cache. Two tasks placed in the same group communicate faster and share cache; tasks placed across groups compete less for those shared resources.

Slurm describes this grouping per node with the Sockets, CoresPerSocket, and ThreadsPerCore settings in slurm.conf. What Slurm treats as a socket is configurable with the per-node Parameters setting:

• numa_node_as_socket – each NUMA domain is a socket.

• l3cache_as_socket – each L3 cache domain is a socket. This can be useful on certain processors where the socket level is too coarse, and the L3 cache may provide better task distribution.

With CPU topology configured, socket-aware scheduling options work as intended: job options such as --ntasks-per-socket and --sockets-per-node, and socket-based allocation with the CR_Socket or CR_Socket_Memory values of the cluster-level SelectTypeParameters setting. For more information about how Slurm allocates sockets, cores, and threads, see Support for Multi-core/Multi-thread Architectures in the Slurm documentation.

AWS PCS doesn't configure CPU topology on Slurm versions before 25.11, so nodes on those versions advertise one core per socket and these options have no effect.

GPU affinity

Each GPU in a node is closest to one NUMA domain, so a subset of the node's cores is local to it. A task that runs on cores local to its GPU transfers data over a shorter path than a task on the other socket. Slurm records this GPU-to-core affinity per device (the Cores attribute of a gres.conf record), along with the GPU-to-GPU interconnect layout (Links).

On AWS PCS compute nodes this information can't be configured statically ahead of time, because device enumeration depends on the individual instance and the devices only become visible when the node boots. Slurm's AutoDetect mechanism solves this: when a node boots, slurmd queries the NVIDIA Management Library (NVML) and reports the detected affinity to the controller at registration.

Note

While a node is powered down, Slurm doesn't know its GPU-to-CPU affinity. The affinity only becomes available when the node boots and registers, so scheduling decisions for powered-down nodes are made without it.

Autodetected GPU affinity is expressed in terms of sockets, so it requires correct CPU topology. If a node registers GPU affinity that doesn't fit the node's configured socket boundaries, Slurm sets the node to INVALID_REG and drains it. For more information, see Troubleshoot invalid node registrations (INVALID_REG) in AWS PCS.

With GPU affinity available, Slurm can place tasks on cores local to their allocated GPUs. Generic Resource (GRES) affinity is handled at socket granularity. Placement depends on the job options; combinations that use the detected affinity include --ntasks-per-socket with --gpus-per-task, and --gpus with --ntasks-per-gpu.

Topology behavior by Slurm version

What AWS PCS configures for you, and what you can configure yourself, depends on the Slurm version of the cluster.

From Slurm 25.11

CPU topology through custom settings

You can configure CPU topology yourself with the Sockets and Parameters custom Slurm settings for a compute node group. For more information, see Custom Slurm settings for AWS PCS compute node groups.

• Set Sockets to the number of domains you want Slurm to treat as sockets. AWS PCS computes and sets the matching CoresPerSocket for you. The value must divide the node's core count evenly.

• Set Parameters to numa_node_as_socket or l3cache_as_socket to declare what the domains are.

With l3cache_as_socket, Sockets is required: AWS PCS can't determine the L3 cache layout of an instance type, so you supply the number of L3 cache domains. This applies on every Slurm version, including 26.05 and later.

Example– L3 cache domains as sockets on an hpc7a.96xlarge compute node group (24 L3 cache domains, 8 cores each)

aws pcs update-compute-node-group \
      --cluster-identifier my-cluster \
      --compute-node-group-identifier my-cng-1 \
      --slurm-configuration \
      'slurmCustomSettings=[{parameterName=Sockets,parameterValue=24},{parameterName=Parameters,parameterValue=l3cache_as_socket}]'

Note

AWS PCS validates the structure of a Sockets value, not its correctness against the hardware. A value that doesn't match the node's real layout leads to suboptimal placement, or to INVALID_REG when GPU autodetection is active.

GPU topology with AutoDetect (opt-in)

On Slurm 25.11, GPU autodetection is off by default. You opt in by adding AutoDetect to the GPU record of the compute node group in gresCustomSettings. For more information, see Configuring custom GRES settings in AWS PCS.

AutoDetect requires the node's CPU topology to be configured. On 25.11, that means setting Sockets for the compute node group as described in the previous section. Without it, nodes register GPU affinity that doesn't fit the default one-core-per-socket layout, and Slurm sets them to INVALID_REG.

From Slurm 26.05

Default CPU topology

From Slurm 26.05, AWS PCS configures NUMA-based CPU topology by default. AWS PCS determines the number of NUMA domains from the instance type of the compute node group, and sets Sockets, CoresPerSocket, and Parameters=numa_node_as_socket on each node.

The following rules apply:

• If a compute node group uses multiple instance types, AWS PCS applies the default topology only when all of them have the same number of NUMA domains. Otherwise, AWS PCS leaves the topology unset, matching the behavior of earlier versions.

• If you set Sockets in the custom Slurm settings of the compute node group, your value takes precedence and AWS PCS doesn't add Parameters. Use this to override the default, for example to use L3 cache domains as sockets with Parameters=l3cache_as_socket, which still requires your Sockets value.

Note

The automatic topology applies to clusters created with Slurm 26.05 or later, and to clusters when you update them to 26.05 or later.

GPU autodetection by default

From Slurm 26.05, AWS PCS enables GPU autodetection by default. AWS PCS adds AutoDetect=full to the GPU record it generates for a GPU compute node group when the node's socket layout is known, either because AWS PCS configured the CPU topology automatically, or because you set Sockets yourself. When the socket layout is not available, AWS PCS emits its GPU record without AutoDetect, matching the behavior of earlier versions. For more information about the GPU record that AWS PCS generates and how it reconciles that record with your own, see How AWS PCS applies your gres.conf records.