Creating a cluster with Slurm accounting

Learn how to configure and create a cluster with Slurm accounting. For more information, see Slurm accounting with AWS ParallelCluster.

When using the AWS ParallelCluster command line interface (CLI) or API, you only pay for the AWS resources that are created when you create or update AWS ParallelCluster images and clusters. For more information, see AWS services used by AWS ParallelCluster.

The PCUI is built on a serverless architecture and you can use it within the AWS Free Tier category for most cases. For more information, see PCUI costs.

In this tutorial, you use a CloudFormation quick-create template (us-east-1) to create an Amazon Aurora for MySQL serverless database. The template instructs CloudFormation to create all the necessary components to deploy an Amazon Aurora serverless database on the same VPC as the cluster. The template also creates a basic networking and security configuration for the connection between the cluster and the database.

Note

Starting with version 3.3.0, AWS ParallelCluster supports Slurm accounting with the cluster configuration parameter SlurmSettings / Database.

Note

The quick-create template serves as an example. This template doesn't cover all possible use cases for a Slurm accounting database server. It's your responsibility to create a database server with the configuration and capacity appropriate for your production workloads.

Prerequisites:

• AWS ParallelCluster is installed.

• The AWS CLI is installed and configured.

• You have an Amazon EC2 key pair.

• You have an IAM role with the permissions that are required to run the pcluster CLI.

• The region that you deploy the quick-create template in supports Amazon Aurora MySQL serverless v2. For more information, see Aurora Serverless v2 with Aurora MySQL.

Step 1: Create the VPC and subnets for AWS ParallelCluster

To use the provided CloudFormation template for the Slurm accounting database, you must have the VPC for the cluster ready. You can do this manually or as part of the Configure and create a cluster with the AWS ParallelCluster command line interface procedure. If you already used AWS ParallelCluster, you might have a VPC ready for the deployment of the cluster and the database server.

Step 2: Create the database stack

Use the CloudFormation quick-create template(us-east-1) to create a database stack for Slurm accounting. The template requires following inputs:

• Database server credentials, specifically the admin user name and password.

• Sizing of the Amazon Aurora serverless cluster. This depends on the expected cluster loading.

• Networking parameters, specifically the target VPC and subnets or CIDR blocks for the creation of the subnets.

Select appropriate credentials and size for your database server. For the networking options, you're required to use the same VPC that the AWS ParallelCluster cluster is deployed to. You can create the subnets for the database and pass them as input to the template. Or, provide two disjoint CIDR blocks for the two subnets and let the CloudFormation template create the two subnets for CIDR blocks. Make sure that the CIDR blocks don't overlap with existing subnets. If the CIDR blocks overlap with existing subnets, the stack fails to be created.

The database server takes several minutes to be created.

Step 3: Create a cluster with Slurm accounting enabled

The provided CloudFormation template generates a CloudFormation stack with some defined outputs. From the AWS Management Console, you can view the outputs in the Outputs tab in the CloudFormation stack view. To enable the Slurm accounting, some of these outputs must be used in the AWS ParallelCluster cluster configuration file:

• DatabaseHost: Used for the SlurmSettings / Database / Uri cluster config parameter.

• DatabaseAdminUser: Used for the SlurmSettings / Database / UserName cluster configuration parameter value.

• DatabaseSecretArn: Used for the SlurmSettings / Database / PasswordSecretArn cluster config parameter.

• DatabaseClientSecurityGroup: This is the security group that's attached to the head node of the cluster that's defined in the HeadNode / Networking / SecurityGroups configuration parameter.

Update your cluster configuration file Database parameters with the output values. Use the pcluster CLI to create the cluster.

$ pcluster create-cluster -n cluster-3.x -c path/to/cluster-config.yaml

After the cluster is created, you can start using Slurm accounting commands such as sacctmgr or sacct.