AWS ParallelCluster API - AWS ParallelCluster
AWS ParallelCluster API DocumentationDeploy with AWS CLIUpdating the APIInvoking AWS ParallelCluster APIAccessing the API logs and metrics

AWS ParallelCluster API

What is AWS ParallelCluster API?

AWS ParallelCluster API is a serverless application that, once deployed to your AWS account, provides programmatic access to AWS ParallelCluster features through an API.

AWS ParallelCluster API is distributed as a self-contained AWS CloudFormation template that includes an Amazon API Gateway endpoint, that exposes AWS ParallelCluster features, and an AWS Lambda function, that takes care of processing the invoked features.

The following image shows a high-level architecture diagram of the AWS ParallelCluster API infrastructure.

AWS ParallelCluster API Documentation

The OpenAPI specification file describing the AWS ParallelCluster API can be downloaded from: 

https://<REGION>-aws-parallelcluster.s3.<REGION>.amazonaws.com/parallelcluster/<VERSION>/api/ParallelCluster.openapi.yaml

Starting from the OpenAPI specification file, you can generate documentation for the AWS ParallelCluster API by using one of the many available tools such as Swagger UI or Redoc.

How to deploy AWS ParallelCluster API

To deploy AWS ParallelCluster API you need to be an Administrator of the AWS account.

The template used to deploy the API is available at the following URL:

https://<REGION>-aws-parallelcluster.s3.<REGION>.amazonaws.com/parallelcluster/<VERSION>/api/parallelcluster-api.yaml

where <REGION> is the AWS Region where the API needs to be deployed to and <VERSION> is the AWS ParallelCluster version (e.g. 3.7.0).

AWS Lambda processes the API invoked features by using a Lambda layer interface with the AWS ParallelCluster Python library API.

Warning

Any user in the AWS account, that has privileged access to AWS Lambda or Amazon API Gateway services, automatically inherits permissions to administer AWS ParallelCluster API resources.

Deploy with AWS CLI

Configure AWS Credentials to be used with the CLI if you haven't already done so.

$ aws configure

Run the following commands to deploy the API:

$ REGION=<region>
$ API_STACK_NAME=<stack-name>  # This can be any name
$ VERSION=3.7.0
$ aws cloudformation create-stack \
    --region ${REGION} \
    --stack-name ${API_STACK_NAME} \
    --template-url https://${REGION}-aws-parallelcluster.s3.${REGION}.amazonaws.com/parallelcluster/${VERSION}/api/parallelcluster-api.yaml \
    --capabilities CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND
$ aws cloudformation wait stack-create-complete --stack-name ${API_STACK_NAME} --region ${REGION}

Customize your deployment

You can customize the API deployment by using the AWS CloudFormation parameters exposed by the template. To configure the value of a parameter when deploying through the CLI, the following option can be used: --parameters ParameterKey=KeyName,ParameterValue=Value.

The following parameters optional:

  • Region - Use the Region parameter to specify whether the API is able to control resources in all AWS Regions (default) or in a single AWS Region. Set this value to the AWS Region the API is being deployed to in order to restrict access.

  • ParallelClusterFunctionRole - This overrides the IAM role that gets assigned to the AWS Lambda function implementing AWS ParallelCluster features. The parameter accepts the ARN of an IAM role. Such role needs to be configured to have AWS Lambda as the IAM principal.

  • CustomDomainName, CustomDomainCertificate, CustomDomainHostedZoneId - Use these parameters to set a custom domain for the Amazon API Gateway endpoint. CustomDomainName is the name of the domain to use, CustomDomainCertificate is the ARN of an AWS managed certificate for this domain name and CustomDomainHostedZoneId is the ID of the Amazon Route 53 hosted zone that you want to create records in.

    Warning

    You can configure custom domain settings to enforce a minimum version of Transport Layer Security (TLS) for the API. For more information, see Choosing a minimum TLS version for a custom domain in API Gateway.

  • EnableIamAdminAccess - By default the AWS Lambda function processing AWS ParallelCluster API operations is configured with an IAM role that prevents any privileged IAM access (EnableIamAdminAccess=false). This makes the API unable to process operations that require the creation of IAM roles or policies. Because of this, the creation of clusters or custom images is successful only when IAM roles are provided as input as part of the resource configuration.

    When EnableIamAdminAccess is set to true the AWS ParallelCluster API is granted permissions to manage the creation of IAM roles required to deploy clusters or generating custom AMIs.

    Warning

    Setting this to true grants IAM admin privileges to the AWS Lambda function processing AWS ParallelCluster operations.

    Refer to AWS ParallelCluster user example policies for managing IAM resources for additional details on the features that can be unlocked when enabling this mode.

  • PermissionsBoundaryPolicy - This optional parameter accepts an existing IAM policy ARN that will be set as permissions boundary for all the IAM roles created by the PC API infrastructure and as a condition on the administrative IAM permissions so that only roles with this policy can be created by the PC API.

    Refer to PermissionsBoundary mode for additional details on the restrictions imposed by this mode.

  • CreateApiUserRole - By default, the deployment of the AWS ParallelCluster API includes the creation of an IAM role which is set as the only role authorized to invoke the API. The Amazon API Gateway endpoint is configured with a resource based policy to grant invoke permission to the created user only. To change this, set CreateApiUserRole=false and then grant API access to selected IAM users. For more information, see Control access for invoking an API in the Amazon API Gateway Developer Guide.

    Warning

    When CreateApiUserRole=true access to the API endpoint is not restricted by Amazon API Gateway resource policies,  all IAM roles having unconstrained execute-api:Invoke permission can access AWS ParallelCluster features. For more information, see Controlling access to an API with API Gateway resource policies in the API Gateway Developer Guide.

    Warning

    The ParallelClusterApiUserRole has permission to invoke all AWS ParallelCluster API operations. To restrict access to a subset of API resources, see the Control who can call an API Gateway API method with IAM policies in the API Gateway Developer Guide.

  • IAMRoleAndPolicyPrefix - This optional parameter accepts a max 10 characters string that will be used as prefix for both IAM roles and policies created as part of the PC API infrastructure.

Updating the API

Upgrading to a newer AWS ParallelCluster version

Option 1: Remove the existing API by deleting the corresponding AWS CloudFormation stack and deploying the new API as shown above.

Option 2: Update the existing API by running the following commands: 

$ REGION=<region>
$ API_STACK_NAME=<stack-name>  # This needs to correspond to the existing API stack name
$ VERSION=3.7.0
$ aws cloudformation update-stack \
    --region ${REGION} \
    --stack-name ${API_STACK_NAME} \
    --template-url https://${REGION}-aws-parallelcluster.s3.${REGION}.amazonaws.com/parallelcluster/${VERSION}/api/parallelcluster-api.yaml \
    --capabilities CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND
$ aws cloudformation wait stack-update-complete --stack-name ${API_STACK_NAME} --region ${REGION}

Invoking AWS ParallelCluster API

The AWS ParallelCluster Amazon API Gateway endpoint is configured with AWS_IAM authorization type, and requires all requests to be SigV4 signed with valid IAM credentials (API reference: making http requests).

When deployed with default settings, API invoke permissions are only granted to the default IAM user created with the API.

To retrieve the ARN of the default IAM user, run: 

$ REGION=<region>
$ API_STACK_NAME=<stack-name>
$ aws cloudformation describe-stacks --region ${REGION} --stack-name ${API_STACK_NAME} --query "Stacks[0].Outputs[?OutputKey=='ParallelClusterApiUserRole'].OutputValue" --output text

To obtain temporary credentials for the default IAM user, run the STS AssumeRole command.

You can retrieve the AWS ParallelCluster API endpoint by running the following command: 

$ REGION=<region>
$ API_STACK_NAME=<stack-name>
$ aws cloudformation describe-stacks --region ${REGION} --stack-name ${API_STACK_NAME} --query "Stacks[0].Outputs[?OutputKey=='ParallelClusterApiInvokeUrl'].OutputValue" --output text

The AWS ParallelCluster API can be invoked by any HTTP client that complies with the OpenAPI specifications that can be found here: 

https://<REGION>-aws-parallelcluster.s3.<REGION>.amazonaws.com/parallelcluster/<VERSION>/api/ParallelCluster.openapi.yaml

Requests need to be SigV4 signed as documented here.

At this time, we do not offer any official API client implementation. However, API clients can be easily generated from the OpenAPI model by using OpenAPI Generator. Once the client is generated SigV4 signing needs to be added if not provided out of the box.

A reference implementation for a Python API client can be found in the AWS ParallelCluster repository. To learn more about how you can use the Python API client, see the Using the AWS ParallelCluster API tutorial.

To implement more advanced access control mechanisms, such as Amazon Cognito or Lambda Authorizers, or to further protect the API with AWS WAF or API keys, follow the Amazon API Gateway documentation.

Warning

An IAM user that is authorized to invoke the AWS ParallelCluster API, can indirectly control all AWS resources managed by AWS ParallelCluster in the AWS account. This includes the creation of AWS resources that the user can't control directly due to restrictions on the user IAM policy. For example, the creation of a AWS ParallelCluster cluster, depending on its configuration, might include the deployment of Amazon EC2 instances, Amazon Route 53, Amazon Elastic File System file systems, Amazon FSx file systems, IAM roles, and resources from other AWS services used by AWS ParallelCluster that the user might not have direct control over.

Warning

When creating a cluster with AdditionalIamPolicies specified in the configuration, the additional policies must match one of the following patterns:

- !Sub arn:${AWS::Partition}:iam::${AWS::AccountId}:policy/parallelcluster*
- !Sub arn:${AWS::Partition}:iam::${AWS::AccountId}:policy/parallelcluster/*
- !Sub arn:${AWS::Partition}:iam::aws:policy/CloudWatchAgentServerPolicy
- !Sub arn:${AWS::Partition}:iam::aws:policy/AmazonSSMManagedInstanceCore
- !Sub arn:${AWS::Partition}:iam::aws:policy/AWSBatchFullAccess
- !Sub arn:${AWS::Partition}:iam::aws:policy/AmazonS3ReadOnlyAccess
- !Sub arn:${AWS::Partition}:iam::aws:policy/service-role/AWSBatchServiceRole
- !Sub arn:${AWS::Partition}:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
- !Sub arn:${AWS::Partition}:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
- !Sub arn:${AWS::Partition}:iam::aws:policy/service-role/AmazonEC2SpotFleetTaggingRole
- !Sub arn:${AWS::Partition}:iam::aws:policy/EC2InstanceProfileForImageBuilder
- !Sub arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

If you need other additional policies, you can do one of the following:

  • Edit the DefaultParallelClusterIamAdminPolicy in:

    https://<REGION>-aws-parallelcluster.s3.<REGION>.amazonaws.com/parallelcluster/<VERSION>/api/parallelcluster-api.yaml

    Add the policy in the ArnLike/iam:PolicyARN section.

  • Omit specifying policies for AdditionalIamPolicies in the configuration file and manually add policies to the AWS ParallelCluster Instance Role created within the cluster.

Accessing the API logs and metrics

API logs are published to Amazon CloudWatch with a retention of 30 days. To retrieve the LogGroup name associated with an API deployment, run the following command: 

$ REGION=<region>
$ API_STACK_NAME=<stack-name>
$ aws cloudformation describe-stacks --region ${REGION} --stack-name ${API_STACK_NAME} --query "Stacks[0].Outputs[?OutputKey=='ParallelClusterLambdaLogGroup'].OutputValue" --output text

Lambda metrics, logs and AWS X-Ray trace logs can be also accessed through the Lambda console. To retrieve the ARN of the Lambda function associated with an API deployment run the following command: 

$ REGION=<region>
$ API_STACK_NAME=<stack-name>
$ aws cloudformation describe-stacks --region ${REGION} --stack-name ${API_STACK_NAME} --query "Stacks[0].Outputs[?OutputKey=='ParallelClusterLambdaArn'].OutputValue" --output text