Slurm guide for multiple queue mode - AWS ParallelCluster

Slurm guide for multiple queue mode

Here you can learn how AWS ParallelCluster and Slurm manage queue (partition) nodes and how you can monitor the queue and node states.

Overview

The scaling architecture is based on Slurm’s Cloud Scheduling Guide and power saving plugin. For more information about the power saving plugin, see Slurm Power Saving Guide. In the architecture, resources that can potentially be made available for a cluster are typically predefined in the Slurm configuration as cloud nodes.

Cloud node lifecycle

Throughout their lifecycle, cloud nodes enter several if not all of the following states: POWER_SAVING, POWER_UP (pow_up), ALLOCATED (alloc), and POWER_DOWN (pow_dn). In some cases, a cloud node might enter the OFFLINE state. The following list details several aspects of these states in the cloud node lifecycle.

  • A node in a POWER_SAVING state appears with a ~ suffix (for example idle~) in sinfo. In this state, no EC2 instances are backing the node. However, Slurm can still allocate jobs to the node.

  • A node transitioning to a POWER_UP state appears with a # suffix (for example idle#) in sinfo. A node automatically transitions to a POWER_UP state, when Slurm allocates a job to a node in a POWER_SAVING state.

    Alternatively, you can transition the nodes to the POWER_UP state manually as an su root user with the command:

    $ scontrol update nodename=nodename state=power_up

    In this stage, the ResumeProgram is invoked, EC2 instances are launched and configured, and the node transitions to the POWER_UP state.

  • A node that is currently available for use appears without a suffix (for example idle) in sinfo. After the node is set up and has joined the cluster, it becomes available to run jobs. In this stage, the node is properly configured and ready for use.

    As a general rule, we recommend that the number of Amazon EC2 instances be the same as the number of available nodes. In most cases, static nodes are available after the cluster is created.

  • A node that is transitioning to a POWER_DOWN state appears with a % suffix (for example idle%) in sinfo. Dynamic nodes automatically enter the POWER_DOWN state after ScaledownIdletime. In contrast, static nodes in most cases aren't powered down. However, you can place the nodes in the POWER_DOWN state manually as an su root user with the command:

    $ scontrol update nodename=nodename state=down reason="manual draining"

    In this state, the instances associated with a node are terminated, and the node is set back to the POWER_SAVING state and available for use after ScaledownIdletime.

    The ScaledownIdletime setting is saved to the Slurm configuration SuspendTimeout setting.

  • A node that is offline appears with a * suffix (for example down*) in sinfo. A node goes offline if the Slurm controller can't contact the node or if the static nodes are disabled and the backing instances are terminated.

Consider the node states shown in the following sinfo example.

$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST efa up infinite 4 idle~ efa-dy-efacompute1-[1-4] efa up infinite 1 idle efa-st-efacompute1-1 gpu up infinite 1 idle% gpu-dy-gpucompute1-1 gpu up infinite 9 idle~ gpu-dy-gpucompute1-[2-10] ondemand up infinite 2 mix# ondemand-dy-ondemandcompute1-[1-2] ondemand up infinite 18 idle~ ondemand-dy-ondemandcompute1-[3-10],ondemand-dy-ondemandcompute2-[1-10] spot* up infinite 13 idle~ spot-dy-spotcompute1-[1-10],spot-dy-spotcompute2-[1-3] spot* up infinite 2 idle spot-st-spotcompute2-[1-2]

The spot-st-spotcompute2-[1-2] and efa-st-efacompute1-1 nodes already have backing instances set up and are available for use. The ondemand-dy-ondemandcompute1-[1-2] nodes are in the POWER_UP state and should be available within a few minutes. The gpu-dy-gpucompute1-1 node is in the POWER_DOWN state, and it transitions into POWER_SAVING state after ScaledownIdletime (defaults to 10 minutes).

All of the other nodes are in POWER_SAVING state with no EC2 instances backing them.

Working with an available node

An available node is backed by an Amazon EC2 instance. By default, the node name can be used to directly SSH into the instance (for example ssh efa-st-efacompute1-1). The private IP address of the instance can be retrieved using the command:

$ scontrol show nodes nodename

Check for IP address in the returned NodeAddr field.

For nodes that aren't available, the NodeAddr field shouldn't point to a running Amazon EC2 instance. Rather, it should be the same as the node name.

Job states and submission

Jobs submitted in most cases are immediately allocated to nodes in the system, or placed in pending if all the nodes are allocated.

If nodes allocated for a job include any nodes in a POWER_SAVING state, the job starts out with a CF, or CONFIGURING state. At this time, the job waits for the nodes in the POWER_SAVING state to transition to the POWER_UP state and become available.

After all nodes allocated for a job are available, the job enters the RUNNING (R) state.

By default, all jobs are submitted to the default queue (known as a partition in Slurm). This is signified by a * suffix after the queue name. You can select a queue using the -p job submission option.

All nodes are configured with the following features, which can be used in job submission commands:

  • An instance type (for example c5.xlarge)

  • A node type (This is either dynamic or static.)

You can see the features for a particular node by using the command:

$ scontrol show nodes nodename

In the return, check the AvailableFeatures list.

Consider the initial state of the cluster, which you can view by running the sinfo command.

$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST efa up infinite 4 idle~ efa-dy-efacompute1-[1-4] efa up infinite 1 idle efa-st-efacompute1-1 gpu up infinite 10 idle~ gpu-dy-gpucompute1-[1-10] ondemand up infinite 20 idle~ ondemand-dy-ondemandcompute1-[1-10],ondemand-dy-ondemandcompute2-[1-10] spot* up infinite 13 idle~ spot-dy-spotcompute1-[1-10],spot-dy-spotcompute2-[1-3] spot* up infinite 2 idle spot-st-spotcompute2-[1-2]

Note that spot is the default queue. It is indicated by the * suffix.

Submit a job to one static node in the default queue (spot).

$ sbatch --wrap "sleep 300" -N 1 -C static

Submit a job to one dynamic node in the EFA queue.

$ sbatch --wrap "sleep 300" -p efa -C dynamic

Submit a job to eight (8) c5.2xlarge nodes and two (2) t2.xlarge nodes in the ondemand queue.

$ sbatch --wrap "sleep 300" -p ondemand -N 10 -C "[c5.2xlarge*8&t2.xlarge*2]"

Submit a job to one GPU node in the gpu queue.

$ sbatch --wrap "sleep 300" -p gpu -G 1

Consider the state of the jobs using the squeue command.

$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 12 ondemand wrap ubuntu CF 0:36 10 ondemand-dy-ondemandcompute1-[1-8],ondemand-dy-ondemandcompute2-[1-2] 13 gpu wrap ubuntu CF 0:05 1 gpu-dy-gpucompute1-1 7 spot wrap ubuntu R 2:48 1 spot-st-spotcompute2-1 8 efa wrap ubuntu R 0:39 1 efa-dy-efacompute1-1

Jobs 7 and 8 (in the spot and efa queues) are already running (R). Jobs 12 and 13 are still configuring (CF), probably waiting for instances to become available.

# Nodes states corresponds to state of running jobs $ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST efa up infinite 3 idle~ efa-dy-efacompute1-[2-4] efa up infinite 1 mix efa-dy-efacompute1-1 efa up infinite 1 idle efa-st-efacompute1-1 gpu up infinite 1 mix~ gpu-dy-gpucompute1-1 gpu up infinite 9 idle~ gpu-dy-gpucompute1-[2-10] ondemand up infinite 10 mix# ondemand-dy-ondemandcompute1-[1-8],ondemand-dy-ondemandcompute2-[1-2] ondemand up infinite 10 idle~ ondemand-dy-ondemandcompute1-[9-10],ondemand-dy-ondemandcompute2-[3-10] spot* up infinite 13 idle~ spot-dy-spotcompute1-[1-10],spot-dy-spotcompute2-[1-3] spot* up infinite 1 mix spot-st-spotcompute2-1 spot* up infinite 1 idle spot-st-spotcompute2-2

Node state and features

In most cases, node states are fully managed by AWS ParallelCluster according to the specific processes in the cloud node lifecycle described earlier in this topic.

However, AWS ParallelCluster also replaces or terminates unhealthy nodes in DOWN and DRAINED states and nodes that have unhealthy backing instances. For more information, see clustermgtd.

Partition states

AWS ParallelCluster supports the following partition states. A Slurm partition is a queue in AWS ParallelCluster.

  • UP: Indicates that the partition is in an active state. This is the default state of a partition. In this state, all nodes in the partition are active and available for use.

  • INACTIVE: Indicates that the partition is in the inactive state. In this state, all instances backing nodes of an inactive partition are terminated. New instances aren't launched for nodes in an inactive partition.

pcluster update-compute-fleet

  • Stopping the compute fleet - When the following command is executed, all partitions transition to the INACTIVE state, and AWS ParallelCluster processes keep the partitions in the INACTIVE state.

    $ pcluster update-compute-fleet --cluster-name testSlurm \ --region eu-west-1 --status STOP_REQUESTED
  • Starting the compute fleet - When the following command is executed, all partitions initially transition to the UP state. However, AWS ParallelCluster processes don't keep the partition in an UP state. You need to change partition states manually. All static nodes become available after a few minutes. Note that setting a partition to UP doesn't power up any dynamic capacity.

    $ pcluster update-compute-fleet --cluster-name testSlurm \ --region eu-west-1 --status START_REQUESTED

When update-compute-fleet is run, you can check the state of the cluster by running the pcluster describe-compute-fleet command and checking the Status. The following lists possible states:

  • STOP_REQUESTED: The stop compute fleet request is sent to the cluster.

  • STOPPING: The pcluster process is currently stopping the compute fleet.

  • STOPPED: The pcluster process finished the stopping process, all partitions are in INACTIVE state, and all compute instances are terminated.

  • START_REQUESTED: The start compute fleet request is sent to the cluster.

  • STARTING: The pcluster process is currently starting the cluster.

  • RUNNING: The pcluster process finished the starting process, all partitions are in the UP state, and static nodes are available after a few minutes.

  • PROTECTED: This status indicates that some partitions have consistent bootstrap failures. Affected partitions are inactive. Please investigate the issue and then run update-compute-fleet to re-enable the fleet.

Manual control of queues

In some cases, you might want to have some manual control over the nodes or queue (known as a partition in Slurm) in a cluster. You can manage nodes in a cluster through the following common procedures using the scontrol command.

  • Power up dynamic nodes in POWER_SAVING state

    Run the command as an su root user:

    $ scontrol update nodename=nodename state=power_up

    You can also submit a placeholder sleep 1 job requesting a certain number of nodes and then rely on Slurm to power up the required number of nodes.

  • Power down dynamic nodes before ScaledownIdletime

    We recommend that you set dynamic nodes to DOWN as an su root user with the command:

    $ scontrol update nodename=nodename state=down reason="manually draining"

    AWS ParallelCluster automatically terminates and resets the downed dynamic nodes.

    In general, we don't recommend that you set nodes to POWER_DOWN directly using the scontrol update nodename=nodename state=power_down command. This is because AWS ParallelCluster automatically handles the power down process.

  • Disable a queue (partition) or stop all static nodes in specific partition

    Set a specific queue to INACTIVE as an su root user with the command:

    $ scontrol update partition=queuename state=inactive

    Doing this terminates all instances backing nodes in the partition.

  • Enable a queue (partition)

    Set a specific queue to UP an su root user with the command:

    $ scontrol update partition=queuename state=up

Scaling behavior and adjustments

Here is an example of the normal scaling workflow:
  • The scheduler receives a job that requires two nodes.

  • The scheduler transitions two nodes to a POWER_UP state, and calls ResumeProgram with the node names (for example queue1-dy-spotcompute1-[1-2]).

  • ResumeProgram launches two Amazon EC2 instances and assigns the private IP addresses and hostnames of queue1-dy-spotcompute1-[1-2], waiting for ResumeTimeout (the default period is 30 minutes before resetting the nodes.

  • Instances are configured and join the cluster. A job starts running on instances.

  • The job completes and stops running.

  • After the configured SuspendTime has elapsed (which is set to ScaledownIdletime), the scheduler sets the instances to the POWER_SAVING state. The scheduler then sets queue1-dy-spotcompute1-[1-2] to the POWER_DOWN state and calls SuspendProgram with the node names.

  • SuspendProgram is called for two nodes. Nodes remain in the POWER_DOWN state, for example, by remaining idle% for a SuspendTimeout (the default period is 120 seconds (2 minutes)). After clustermgtd detects that nodes are powering down, it terminates the backing instances. Then, it transitions queue1-dy-spotcompute1-[1-2] to the idle state and resets the private IP address and hostname so it is ready to power up for future jobs.

If things go wrong and an instance for a particular node can't be launched for some reason, then the following happens:
  • The scheduler receives a job that requires two nodes.

  • The scheduler transitions two cloud bursting nodes to the POWER_UP state and calls ResumeProgram with the nodenames, (for example queue1-dy-spotcompute1-[1-2]).

  • ResumeProgram launches only one (1) Amazon EC2 instance and configures queue1-dy-spotcompute1-1, with one (1) instance, queue1-dy-spotcompute1-2, failing to launch.

  • queue1-dy-spotcompute1-1 isn't impacted and comes online after reaching the POWER_UP state.

  • queue1-dy-spotcompute1-2 transitions to the POWER_DOWN state, and the job is requeued automatically because Slurm detects a node failure.

  • queue1-dy-spotcompute1-2 becomes available after SuspendTimeout (the default is 120 seconds (2 minutes)). In the meantime, the job is requeued and can start running on another node.

  • The above process repeats until the job can run on an available node without a failure occurring.

There are two timing parameters that can be adjusted if needed:
  • ResumeTimeout (the default is 30 minutes): ResumeTimeout controls the time Slurm waits before transitioning the node to the down state.

    • It might be useful to extend ResumeTimeout if your pre/post installation process takes nearly that long.

    • ResumeTimeout is also the maximum time that AWS ParallelCluster waits before replacing or resetting a node if there is an issue. Compute nodes self-terminate if any error occurs during launch or setup. AWS ParallelCluster processes replace a node upon detection of a terminated instance.

  • SuspendTimeout (the default is 120 seconds (2 minutes)): SuspendTimeout controls how quickly nodes get placed back into the system and are ready for use again.

    • A shorter SuspendTimeout means that nodes are reset more quickly, and Slurm can try to launch instances more frequently.

    • A longer SuspendTimeout means that failed nodes are reset more slowly. In the meantime, Slurm tries to use other nodes. If SuspendTimeout is more than a few minutes, Slurm tries to cycle through all nodes in the system. A longer SuspendTimeout might be beneficial for large-scale systems (over1,000 nodes) to reduce stress on Slurm when it tries to frequently re-queue failing jobs.

    • Note that SuspendTimeout doesn't refer to the time AWS ParallelCluster waits to terminate a backing instance for a node. Backing instances for POWER_DOWN nodes are immediately terminated. The terminate process usually is finished in a few minutes. However, during this time, the node remains in the POWER_DOWN state and isn't available for the scheduler's use.

Logs for the architecture

The following list contains the key logs. The log stream name used with Amazon CloudWatch Logs has the format {hostname}.{instance_id}.{logIdentifier}, where logIdentifier follows the log names.

  • ResumeProgram: /var/log/parallelcluster/slurm_resume.log (slurm_resume)

  • SuspendProgram: /var/log/parallelcluster/slurm_suspend.log (slurm_suspend)

  • clustermgtd: /var/log/parallelcluster/clustermgtd.log (clustermgtd)

  • computemgtd: /var/log/parallelcluster/computemgtd.log (computemgtd)

  • slurmctld: /var/log/slurmctld.log (slurmctld)

  • slurmd: /var/log/slurmd.log (slurmd)

Common issues and how to debug:

Nodes that failed to launch, power up, or join the cluster

  • Dynamic nodes:

    • Check the ResumeProgram log to see if ResumeProgram was called with the node. If not, check the slurmctld log to determine if Slurm tried to call ResumeProgram with the node. Note that incorrect permissions on ResumeProgram might cause it to fail silently.

    • If ResumeProgram is called, check to see if an instance was launched for the node. If the instance didn't launch, there should be clear error message as to why the instance failed to launch.

    • If an instance was launched, there may have been some problem during the bootstrap process. Find the corresponding private IP address and instance ID from the ResumeProgram log and look at corresponding bootstrap logs for the specific instance in CloudWatch Logs.

  • Static nodes:

    • Check the clustermgtd log to see if instances were launched for the node. If instances didn't launch, there should be clear errors on why the instances failed to launch.

    • If an instance was launched, there is some problem with the bootstrap process. Find the corresponding private IP and instance ID from the clustermgtd log and look at corresponding bootstrap logs for the specific instance in CloudWatch Logs.

Nodes replaced or terminated unexpectedly, and node failures

  • Nodes replaced/terminated unexpectedly:

    • In most cases, clustermgtd handles all node maintenance actions. To check if clustermgtd replaced or terminated a node, check the clustermgtd log.

    • If clustermgtd replaced or terminated the node, there should be a message indicating the reason for the action. If the reason is scheduler related (for example, the node was DOWN), check in the slurmctld log for more details. If the reason is Amazon EC2 related, use tools such as Amazon CloudWatch or the Amazon EC2 console, CLI, or SDKs, to check status or logs for that instance. For example, you can check if the instance had scheduled events or failed Amazon EC2 health status checks.

    • If clustermgtd didn't terminate the node, check if computemgtd terminated the node or if EC2 terminated the instance to reclaim a Spot Instance.

  • Node failures:

    • In most cases, jobs are automatically requeued if a node failed. Look in the slurmctld log to see why a job or a node failed and assess the situation from there.

Failure when replacing or terminating instances, failure when powering down nodes

  • In general, clustermgtd handles all expected instance termination actions. Look in the clustermgtd log to see why it failed to replace or terminate a node.

  • For dynamic nodes failing ScaledownIdletime, look in the SuspendProgram log to see if slurmctld processes made calls with the specific node as argument. Note SuspendProgram doesn't actually perform any specific action. Rather, it only logs when it’s called. All instance termination and NodeAddr resets are completed by clustermgtd. Slurm transitions nodes to IDLE after SuspendTimeout.

Other issues:

  • AWS ParallelCluster doesn't make job allocation or scaling decisions. It only tries to launch, terminate, and maintain resources according to Slurm’s instructions.

    For issues regarding job allocations, node allocation and scaling decision, look at the slurmctld log for errors.