Deployment using custom installer - Scale-Out Computing on AWS
Automated deployment

Deployment using custom installer

For customers who want to leverage existing AWS resources required for the solution while maintaining customization and extensibility, this solution allows you to deploy the AWS CloudFormation template using the AWS Cloud Development Kit (AWS CDK) based custom installer in your hosted repository for production environments. The CDK based installer also allows customization to all resources created during the installation. For example, you can choose how many NAT Gateways to deploy (default to 1), the KMS encryption to use for your file systems (default to aws/key), the instance type (default to m5.large) to provision for the scheduler, and other options that are specified with the CloudFormation implementation.

Note

These instructions are applicable to versions 2.7.0 or later of this AWS Solution. For older versions, refer to the legacy documentation.

This solution is a collection of AWS CloudFormation templates, Amazon Elastic Compute Cloud (Amazon EC2) user data bootstrap scripts, and Python scripts. Before deploying, create a custom installer based on the build ID you want to use.

Automated deployment

Before you launch the automated deployment, please review the architecture, prerequisites, and other considerations discussed in this guide. Follow the step-by-step instructions in this section to deploy this solution into your account using a custom installer.

Time to deploy: Approximately 60 minutes

Deployment overview

Use the following steps to deploy this solution on AWS. For detailed instructions, follow the links for each step.

Prerequisites

  • Set up an Amazon S3 bucket.

Step 1. Create and confirm required IAM policies

  • Apply IAM policies required to deploy the solution

Step 2. Download the solution template

  • Download the solution from the Github repository

Step 3. Create and run the installer

  • Create and upload the build

  • Review solution parameters, and adjust if necessary

  • Launch the solution installer

Prerequisites

Set up Amazon S3 bucket

This solution uses an Amazon S3 bucket for storing data. Before deploying this solution, you must create a new Amazon S3 bucket in your AWS account. Or, you can use an existing Amazon S3 bucket.

Step 1. Create and confirm required IAM policies

Note

Following step 2, you can find the list of all required IAM policies to install the solution via installer/SOCAInstallerIamPolicy.json

If needed, you can create an IAM policy and assign it to the use or Role you are planning to use to install this solution.

  1. Navigate to IAM in the AWS console, select Policies in the left sidebar menu then choose Create Policy.

  2. Select JSON and copy and paste the content of installer/SOCAInstallerIamPolicy.json

  3. Select the user or Role you are using to install Scale-Out Computing on AWS.

  4. Choose Add Permissions and attach the policy you just created.

Step 2. Download the solution template

This solution is open-source and hosted on GitHub. Use the following procedure to download the solution template:

  1. Navigate to the GitHub repository, select Clone or download.

  2. Select Download Zip.

  3. When the zip file is downloaded, extract the file on your local machine.

Step 3. Create and run the installer

After you have cloned your repository, run the installer/soca_installer.sh script. The installer will perform the following tasks:

Note

For the build and installer environment, Linux (Amazon Linux 2, RHEL, Centos) or MacOS host operating system are supported to run the solution installer.

  • Check if Python3 is available on your system

  • Create a custom Python virtual-environment and install required libraries

  • Install NodeJS, NPM, AWS CDK and AWS Command Line Interface if needed

  • Set up your Scale-Out Computing on AWS cluster

The installer is built with AWS CDK. To learn more, refer to AWS CDK.

Note

You are responsible for the cost of the AWS services used while running this solution. Refer to the Cost section for more details. For full details, refer to the pricing webpage for each AWS service you will be using in this solution.

  1. Run the soca_install.sh script located in the installer folder. Assuming your current working directory is the root level of the solution stack.

    
$ ./installer/soca_install.sh

  2. The installer script will prompt for your cluster parameters. Follow the instructions and choose a Amazon S3 bucket you own, the name of your cluster, the SSH keypair to use and other cluster parameters.

    Note

    You can pass all parameters via arguments to automate the installation process. Run soca_installer.sh --help to see all options available.

Once all the parameters are specified, installer will run the cdk bootstrap command. This action will create a staging S3 bucket and store all assets generated by AWS CDK. No actions will be performed if you already have your environment activated for AWS CDK.

The installer will upload the scripts (<100 MB) required to configure the scheduler to the Amazon S3 bucket you specified.

Next, the installer will trigger a cdk deploy command and the deployment will start. This will create a new CloudFormation stack on your AWS account.

Once the CloudFormation stack is created, the installer will verify if the solution is configured correctly. The installer will exit once the solution is fully configured and reachable.