AWS ParallelCluster
AWS ParallelCluster User Guide

[cluster] Section

Defines one or more clusters for different job types or workloads.

Each cluster can have its own configuration.

The format is [cluster <clustername>]. The [cluster] section named by the cluster_template setting in the [global] section is used.

[cluster default]

additional_cfn_template

Defines an additional AWS CloudFormation template to launch along with the cluster. This additional template is used for the creation of resources that exist outside of the cluster but are part of the cluster's lifecycle.

When set to a value other than NONE, it must be an HTTP URL to a public template, with all parameters provided.

The default value is NONE.

additional_cfn_template = NONE

additional_iam_policies

Specifies a comma-separated list of Amazon Resource Names (ARNs) of IAM policies for Amazon EC2. This list is attached to the root role used in the cluster, in addition to the permissions required by AWS ParallelCluster. An IAM policy name and its ARN are different. Names cannot be used as an argument to additional_iam_policies. additional_iam_policies should be used instead of the ec2_iam_role. This is because additional_iam_policies are added to the permissions that AWS ParallelCluster requires, and the ec2_iam_role must include all permissions required. The permissions required often change from release to release as features are added.

The default value is NONE.

additional_iam_policies = arn:aws:iam::aws:policy/AdministratorAccess

Note

Support for additional_iam_policies was added in AWS ParallelCluster 2.5.0.

base_os

Specifies which OS type is used in the cluster.

Available options are:

  • alinux

  • centos6

  • centos7

  • ubuntu1604

  • ubuntu1804

Note

Support for ubuntu1804 was added and support for ubuntu1404 was removed in AWS ParallelCluster 2.5.0.

Supported operating systems by Region are listed in the following table. Note that "commercial" entails all other supported Regions including us-east-1, us-west-2, and so on.

Partition (Regions) alinux centos6 centos7 ubuntu1604 ubuntu1804
Commercial (All Regions not mentioned below) True True True True True
AWS GovCloud (US-East) (us-gov-east-1) True False False True True
AWS GovCloud (US-West) (us-gov-west-1) True False False True True
China (Beijing) (cn-north-1) True False False True True
China (Ningxia) (cn-northwest-1) True False False True True

Note: The base_os parameter also determines the user name that is used to log into the cluster.

  • centos6 and centos7: centos

  • ubuntu1604, and ubuntu1804: ubuntu

  • alinux: ec2-user

The default value is alinux.

base_os = alinux

cluster_type

Defines the type of cluster to launch.

Valid options are: ondemand, and spot.

The default value is ondemand.

For more information about Spot Instances, see Working with Spot Instances.

cluster_type = ondemand

compute_instance_type

Defines the Amazon EC2 instance type that is used for the cluster compute nodes.

If you are using the awsbatch scheduler, see the Compute Environments creation in the AWS Batch UI for a list of supported instance types.

Defaults to t2.micro, optimal when the scheduler is awsbatch.

compute_instance_type = t2.micro

A1 instances are not supported.

compute_root_volume_size

Specifies the ComputeFleet root volume size in GB. The AMI must support growroot.

The default value is 25.

Note

Prior to AWS ParallelCluster. 2.5.0 the default was 20.

compute_root_volume_size = 20

custom_ami

Specifies the ID of a custom AMI to use instead of the default published AMIs.

The default value is NONE.

custom_ami = NONE

dcv_settings

Identifies the [dcv] section with the NICE DCV configuration.

For more information, see the [dcv] section.

For example, the following setting specifies that the section that starts [dcv custom-dcv] is used for the NICE DCV configuration.

dcv_settings = custom-dcv

Note

Support for dcv_settings was added in AWS ParallelCluster 2.5.0.

desired_vcpus

Specifies the desired number of vCPUs in the compute environment. Used only if the scheduler is awsbatch.

The default value is 4.

desired_vcpus = 4

disable_hyperthreading

Disables hyperthreading on the master and compute nodes. Not all instance types can disable hyperthreading. For a list of instance types that support disabling hyperthreading, see CPU Cores and Threads Per CPU Core Per Instance Type in the Amazon EC2 User Guide for Linux Instances.

disable_hyperthreading = true

Note

Support for disable_hyperthreading was added in AWS ParallelCluster 2.5.0.

ebs_settings

Identifies the [ebs] sections with the Amazon EBS volumes that are mounted on the master instance. When using multiple Amazon EBS volumes, enter these parameters as a comma-separated list.

Up to five (5) additional Amazon EBS volumes are supported.

For more information, see the [ebs] section.

For example, the following setting specifies that the sections that start [ebs custom1] and [ebs custom2] are used for the Amazon EBS volumes.

ebs_settings = custom1, custom2

ec2_iam_role

Defines the name of an existing IAM role for Amazon EC2 that is attached to all instances in the cluster. An IAM role name and its Amazon Resource Name (ARN) are different. ARNs cannot be used as an argument to ec2_iam_role. If this option is specified, the additional_iam_policies setting is ignored. AWS recommends using additional_iam_policies rather than the ec2_iam_role, because features added to AWS ParallelCluster often require new permissions.

The default value is NONE.

ec2_iam_role = NONE

efs_settings

Specifies settings related to the Amazon EFS filesystem.

For more information, see the [efs] section.

For example, the following setting specifies that the section that starts [efs customfs] is used for the Amazon EFS filesystem configuration.

efs_settings = customfs

enable_efa

If present, specifies that Elastic Fabric Adapter (EFA) is enabled for the compute nodes. EFA is supported by specific instance types (c5n.18xlarge, c5n.metal, i3en.24xlarge, p3dn.24xlarge). For more information, see Elastic Fabric Adapter.

enable_efa = compute

enable_intel_hpc_platform

If present, indicates that the End User License Agreement for Intel Parallel Studio is accepted. This will cause Intel Parallel Studio to be installed on the master node and shared with the compute nodes. This adds several minutes to the time it takes the master node to bootstrap.

enable_intel_hpc_platform = true

Note

Support for enable_intel_hpc_platform was added in AWS ParallelCluster 2.5.0.

encrypted_ephemeral

Encrypts the ephemeral instance store volumes with non-recoverable in-memory keys, using LUKS (Linux Unified Key Setup).

For more information, see https://gitlab.com/cryptsetup/cryptsetup/blob/master/README.md.

The default value is false.

encrypted_ephemeral = false

ephemeral_dir

Defines the path where instance store volumes are mounted, if they are used.

The default value is /scratch.

ephemeral_dir = /scratch

extra_json

Defines the extra JSON that is merged into the dna.json that is used by Chef.

The default value is {}.

extra_json = {}

fsx_settings

Specifies the section that defines the Amazon FSx for Lustre configuration.

For more information, see the [fsx] section.

fsx_settings = fs

For example, the following setting specifies that the section that starts [fsx fs] is used for the Amazon FSx for Lustre configuration.

fsx_settings = fs

initial_queue_size

Sets the initial number of Amazon EC2 instances to launch as compute nodes in the cluster.

This setting is applicable only for traditional schedulers (SGE, Slurm, and Torque).

If the scheduler is awsbatch, use min_vcpus instead.

Defaults to 2.

initial_queue_size = 2

key_name

Names an existing Amazon EC2 key pair with which to enable SSH access to the instances.

key_name = mykey

maintain_initial_size

Maintains the initial size of the Auto Scaling group for traditional schedulers (SGE, Slurm, and Torque).

If the scheduler is awsbatch, use desired_vcpus instead.

This setting is a Boolean flag. If set to true, the Auto Scaling group never has fewer members than the value of initial_queue_size. The cluster can still scale up to the value of max_queue_size. If cluster_type = spot then the Auto Scaling group can have instances interrupted and the size can drop below initial_queue_size.

If set to false, the Auto Scaling group can scale down to zero (0) members to prevent resources from sitting idle when they are not needed.

Defaults to false.

maintain_initial_size = false

master_instance_type

Defines the Amazon EC2 instance type that is used for the master node.

Defaults to t2.micro.

master_instance_type = t2.micro

A1 instances are not supported.

master_root_volume_size

Specifies the MasterServer root volume size in GB. The AMI must support growroot.

The default value is 25.

Note

Prior to AWS ParallelCluster. 2.5.0 the default was 20.

master_root_volume_size = 20

max_queue_size

Sets the maximum number of Amazon EC2 instances that can be launched in the cluster.

This setting is applicable only for traditional schedulers (SGE, Slurm, and Torque).

If the scheduler is awsbatch, use max_vcpus instead.

Defaults to 10.

max_queue_size = 10

max_vcpus

Specifies the maximum number of vCPUs in the compute environment. Used only if the scheduler is awsbatch.

The default value is 20.

max_vcpus = 20

min_vcpus

Maintains the initial size of the Auto Scaling group for the awsbatch scheduler.

If the scheduler is SGE, Slurm, or Torque, use maintain_initial_size instead.

The compute environment never has fewer members than the value of min_vcpus.

Defaults to 0.

min_vcpus = 0

placement

Defines the cluster placement group logic, enabling either the whole cluster or only the compute instances to use the cluster placement group.

Valid options are cluster or compute.

This parameter is not used when the scheduler is awsbatch.

The default value is compute.

placement = compute

placement_group

Defines the cluster placement group.

Valid options are:

  • NONE

  • DYNAMIC

  • An existing Amazon EC2 cluster placement group name

When set to DYNAMIC, a unique placement group is created and deleted as part of the cluster stack.

This parameter is not used when the scheduler is awsbatch.

For more information about placement groups, see Placement Groups in the Amazon EC2 User Guide for Linux Instances.

The default value is NONE.

Not all instance types support cluster placement groups. For example, the default instance type of t2.micro does not support cluster placement groups. For information about the list of instance types that support cluster placement groups, see Cluster Placement Group Rules and Limitations in the Amazon EC2 User Guide for Linux Instances. See Placement Groups and Instance Launch Issues for tips when working with placement groups.

placement_group = NONE

post_install

Specifies the URL of a postinstall script that is executed after all of the boot_as_* scripts are run.

When using awsbatch as the scheduler, the postinstall script is executed only on the master node.

The parameter format can be either http://hostname/path/to/script.sh or s3://bucketname/path/to/script.sh.

The default value is NONE.

post_install = NONE

post_install_args

Specifies a quoted list of arguments to pass to the postinstall script.

The default value is NONE.

post_install_args = "NONE"

pre_install

Specifies the URL of a preinstall script that is executed before any of the boot_as_* scripts are run.

When using awsbatch as the scheduler, the preinstall script is executed only on the master node.

The parameter format can be either http://hostname/path/to/script.sh or s3://bucketname/path/to/script.sh.

The default value is NONE.

pre_install = NONE

pre_install_args

Specifies a quoted list of arguments to pass to the preinstall script.

The default value is NONE.

pre_install_args = "NONE"

proxy_server

Defines an HTTP or HTTPS proxy server, typically http://x.x.x.x:8080.

The default value is NONE.

proxy_server = NONE

raid_settings

Identifies the [raid] section with the Amazon EBS volume RAID configuration.

For more information, see the [raid] section.

For example, the following setting specifies that the section that starts [raid rs] be used for the Auto Scaling configuration.

raid_settings = rs

s3_read_resource

Specifies an Amazon S3 resource to which AWS ParallelCluster nodes are granted read-only access.

For example, arn:aws:s3:::my_corporate_bucket/* provides read-only access to all objects in the my_corporate_bucket bucket.

See working with Amazon S3 for details on format.

The default value is NONE.

s3_read_resource = NONE

s3_read_write_resource

Specifies an Amazon S3 resource to which AWS ParallelCluster nodes are granted read/write access.

For example, arn:aws:s3:::my_corporate_bucket/Development/* provides read/write access to all objects in the Development folder of the my_corporate_bucket bucket.

See working with Amazon S3 for details on format.

The default value is NONE.

s3_read_write_resource = NONE

scaling_settings

Identifies the [scaling] section with the Auto Scaling configuration.

For more information, see the [scaling] section.

For example, the following setting specifies that the section that starts [scaling custom] is used for the Auto Scaling configuration.

scaling_settings = custom

scheduler

Defines the cluster scheduler.

Valid options are:

  • sge

  • torque

  • slurm

  • awsbatch

For more information about the awsbatch scheduler, see networking setup.

The default value is sge.

scheduler = sge

shared_dir

Defines the path where the shared Amazon EBS volume is mounted.

Do not use this option with multiple Amazon EBS volumes. Instead, provide shared_dir values under each Amazon EBS [ebs] section.

See the Amazon EBS Section for details on working with multiple Amazon EBS volumes.

The default value is /shared.

The following example shows a shared Amazon EBS volume mounted at /myshared.

shared_dir = myshared

spot_bid_percentage

Optionally sets the on-demand bid percentage used to calculate the maximum Spot price for the ComputeFleet, when awsbatch is the scheduler.

If unspecified, the current spot market price is selected, capped at the On-Demand price.

spot_bid_percentage = 85

spot_price

Optionally sets the maximum Spot price for the ComputeFleet on traditional schedulers (SGE, Slurm, and Torque). Used only when the cluster_type is set to spot. If you do not specify a value, you are charged the Spot price, capped at the On-Demand price.

If the scheduler is awsbatch, use spot_bid_percentage instead.

For assistance finding a bid price that meets your needs, see the Spot Bid Advisor.

spot_price = 1.50

tags

Defines tags to be used by AWS CloudFormation.

If command line tags are specified via --tags, they are merged with config tags.

Command line tags overwrite config tags that have the same key.

Tags are JSON formatted. Do not use quotes outside of the curly braces.

For more information, see AWS CloudFormation Resource Tags Type in the AWS CloudFormation User Guide.

tags = {"key" : "value", "key2" : "value2"}

template_url

Defines the path to the AWS CloudFormation template that is used to create the cluster.

Updates use the template that was originally used to create the stack.

Defaults to https://<aws_region_name>-aws-parallelcluster.s3.amazonaws.com/templates/aws-parallelcluster-<version>.cfn.json.

template_url = https://us-east-1-aws-parallelcluster.s3.amazonaws.com/templates/aws-parallelcluster.cfn.json

vpc_settings

Identifies the [vpc] section with the Amazon VPC configuration where the cluster is deployed.

For more information, see the [vpc] section.

For example, the following setting specifies that the section that starts [vpc public] is used for the Amazon VPC configuration.

vpc_settings = public