Automated deployment - Service Workbench on AWS

Automated deployment

Before you launch the solution, review the architecture, configuration, network security, and other considerations discussed in this guide. Follow the step-by-step instructions in this section to configure and deploy the solution into your account.

Time to deploy: Approximately 45 minutes.

Prerequisites

AWS Organizations

If you wish to enable Service Workbench on AWS to create new AWS accounts via the Create AWS Account feature, then AWS Organizations must be enabled.

Cost Explorer

In order to see any actual cost in dashboards and workspaces, the main account must be set up in Cost Explorer. The main account holds the AWS Organization which creates member accounts.

Deployment overview

Use the following steps to deploy this solution on AWS. For detailed instructions, follow the links for each step.

Step 1. Launch the stack

  • Launch the AWS CloudFormation template into your AWS account.

  • Enter values for required parameters.

  • Review the other template parameters, and adjust if necessary.

Step 2. Post-launch tasks

  • Tasks in the AWS Management Console

  • Tasks in the Service Workbench on AWS user web portal

Step 1. Launch the stack

This automated AWS CloudFormation template deploys Service Workbench on AWS in the AWS Cloud.

Note

You are responsible for the cost of the AWS services used while running this solution. For more details, visit to the Cost section in this guide, and refer to the pricing webpage for each AWS service used in this solution.

  1. Sign in to the AWS Management Console and use the button below to launch the AWS CloudFormation template.

    
              Service Workbench on AWS launch button

    Alternatively, you can download the template as a starting point for your own implementation.

  1. The template launches in the US East (N. Virginia) Region by default. To launch the solution in a different AWS Region, use the Region selector in the console navigation bar.

Note

This solution uses the Amazon Service Catalog service, which is not currently available in all AWS Regions. You must launch this solution in an AWS Region where Amazon Service Catalog is available. For the most current availability by Region, refer to the AWS Service Region Table.

  1. On the Create stack page, verify that the correct template URL is in the Amazon S3 URL text box and choose Next.

  2. On the Specify stack details page, assign a name to your solution stack. For information about naming character limitations, see IAM and STS Limits in the AWS Identity and Access Management User Guide.

  3. Under Parameters, review the parameters for this solution template and modify them as necessary. This solution uses the following default values.

Parameter Default Description
CreateMachineImages true Generate Amazon Machine Images (AMIs) during deployment. See Amazon Machine Images (AMI) for more information.
EnvironmentType dev Enables grouping multiple environments with a common name. Use dev, demo, qa, or prod.
ServicePortfolio true Create Service Catalog product and portfolio entries.
SolutionName swb Included in all resource names created by the deployment. Limited to 7 characters.
StageName test Allows multiple Service Workbench on AWS installations into the same account. Included in resource names created by the deployment. Should be limited to 5 characters.
  1. Choose Next.

  2. On the Configure stack options page, choose Next.

  1. On the Review page, review and confirm the settings. Check the box acknowledging that the template will create AWS Identity and Access Management (IAM) resources.

  1. Choose Create stack to deploy the stack.

    You can view the status of the stack in the AWS CloudFormation Console in the Status column. You should receive a CREATE_COMPLETE status in approximately 20 minutes.

Note

Service Workbench on AWS creates four additional stacks during deployment: Infrastructure, Backend, edgeLambda, and postDeployment.

Step 2. Post-launch tasks

1. Gather information for post-launch tasks

2. Onboard AWS account

3. Configure accounts, users, projects, and indexes

4. Import and test Service Catalog Products

5. Create a research study

Gather information for post-launch tasks

Configuration of Service Workbench on AWS requires output values created during deployment. Use the steps below to gather this information.

  1. Sign in to the AWS Management Console

  2. Navigate to CloudFormation

  1. Click on View stacks

  2. Select the infrastructure stack created by Service Workbench on AWS

  3. Click on Outputs

  4. Note down the WebsiteUrl output.

  5. Select the backend stack created by Service Workbench on AWS

  6. Note down the ApiHandlerRoleArn and WorkflowLoopRunnerRoleArn outputs.

  7. Navigate to AWS Systems Manager

  8. Click on Parameter Store

  9. Select the /user/root/password parameter created by Service Workbench on AWS.

  10. Click Show to reveal the password and note it down.

  11. Navigate to Amazon EC2.

  12. Click on AMIs in the left navigation panel.

  13. Note down the Names and IDs for all AMIs created by Service Workbench on AWS.

Onboard AWS account

Service Workbench on AWS enables users to launch compute resources into an AWS account. Onboarding the AWS account creates the necessary roles and permissions for Service Workbench on AWS to use.

  1. Sign in to the AMS Management Console.

  2. Navigate to CloudFormation.

  1. Download onboard-account.cfn.yml by clicking this link and saving the file.

  2. Click Create stack, and upload the onboard-account.cfn.yml file.

  3. Click Next.

  4. Under Parameters, review the parameters and modify them as necessary.

Parameter

Default Description
Namespace <Requires input> An environment name that will be prefixed to resource names; use the stage name.
CentralAccountId <Requires input> The account number of the AWS account where the solution is deployed.
ExternalId <Requires input> A unique ID used to identify this account; use workbench.
VpcCidr 10.0.0.0/16 IP range (CIDR notation) for this VPC
VpcPublicSubnet1Cidr 10.0.0.0/19 IP range (CIDR notation) for the public subnet in the 1st Availability Zone
ApiHandlerArn <Requires input> The ARN of apiHandler role noted down earlier.
LaunchConstraintPolicyPrefix * Customer managed policy name prefix to use when creating a launch constraint role in the on-boarded account
LaunchConstraintRolePrefix * Role name prefix to use when creating a launch constraint role in the on-boarded account
WorkflowRoleArn <Requires input> The ARN of workflowRunner role
  1. Choose Next.

  2. On the Configure stack options page, choose Next.

  3. On the Review page, review and confirm the settings. Check the box acknowledging that the template will create AWS Identity and Access Management (IAM) resources.

  4. Choose Create stack to deploy the stack.

  5. You can view the status of the stack in the AWS CloudFormation Console in the Status column. You should receive a CREATE_COMPLETE status in approximately 5 minutes.

  6. When the stack reaction is complete, click on Outputs.

  7. Note down the output values for CrossAccountEnvMgmtRoleArn, CrossAccountExecutionRoleArn, EncryptionKeyArn, VPC, and VpcPublicSubnet1.

Configure accounts, users, projects, and indexes

  1. Open a web browser.

  2. Open a web browser and navigate to WebsiteUrl as noted down earlier.

  3. Sign in to Service Workbench on AWS web portal.

    1. Username: root

    2. Password: the /user/root/password as noted down earlier.

Note

Logging in as the root user is not recommended after initial step. The following steps include creating an administrator user for future use.

  1. Add AWS Account to Service Workbench on AWS

    1. Click Accounts in the left navigation pane.

    2. Click AWS Accounts.

    3. Click Add AWS Account.

    4. Provide account details.

Value Notes
Account Name <enter text>
AWS Account ID <enter text> Provide the number of the AWS account that you deployed Service Workbench on AWS into.
Role Arn <enter text> Provide CrossAccountExecutionRoleArn as noted down earlier.
AWS Service Catalog Role Arn <enter text> Provide CrossAccountEnvMgmtRoleArn as noted down earlier.
External ID <enter text> Enter workbench.
Description <enter text>
VPC ID <enter text> Provide VPC as noted down earlier.
Subnet ID <enter text> Provide VpcPublicSubnet1 as noted down earlier.
KMS Encryption Key ARN <enter text> Provide EncryptionKeyArn as noted down earlier.
  1. After adding the account, click Index tab.

Projects and Indexes (cost centers) form a hierarchy under Accounts. Each Account can have multiple Indexes and each Index can have multiple Projects. Projects are attached to Users, so you must create the Projects first.

  1. Click Add Index. See Figure 2.



Create an Index

Figure 2: Create an Index

  1. Provide an Index ID, AWS Account ID, and Description; then click Add Index.

  2. Click Projects tab. See Figure 3.

Figure 3: Create a Project

  1. Click Add Project.

  2. Provide a Project Name, select an Index ID, provide a Description, and select Project Admins; then click Add Project.

  3. Click Users in the left navigation panel.

  4. Click Roles tab. See Figure 4.

Figure 4: Roles tab

Roles are used to control access to Workspace Types in Service Workbench on AWS.

  1. Click Users tab.

  2. Click Add Local User

Note

Service Workbench on AWS supports local users for rapid setup for configuration, evaluation, and testing. For production usage, configure Federated users.

  1. Create an administrator local user.

    1. For UserRole, select admin.

    1. For Status, select Active.

    2. Click Add Local User.

  1. Click Logout in the top right corner of the page.

  2. Log in as the administrative user created in the previous step.

Import and test Service Catalog Products

Service Workbench on AWS uses AWS Service Catalog to manage different types of computation resources available for researchers to use through the platform. Each product can have multiple size configurations defined.

When Service Workbench on AWS is deployed, an AWS Service Catalog portfolio is created with four commonly used products: Amazon SageMaker, Amazon EC2 for Windows, Amazon EC2 for Linux and Amazon EMR. These definitions must be imported into Service Workbench on AWS and configured before they can be deployed.

Access to Workspace Type configurations can be controlled based on the user’s Role.

Note

Creating Workspace Types requires the IDs for the AMIs created during deployment. The AMI IDs are collected the “Tasks in AWS Management Console” section above.

  1. Click Workspace Types in the left navigation bar.

AWS Service Catalog products available for import are listed. See Figure 5.

Figure 5: List of AWS Service Catalog products available for import

  1. Click Import on the AWS Service Catalog product that you wish to import.

  2. Provide Basic information.

  3. Click Import Workspace Type.

  4. Click Add Configuration.

Figure 6: Steps for creating a Workspace Type Configurations

  1. Input Basic Information

    1. Provide a unique ID for the configuration.

    2. Provide a Name for the configuration.

    3. Provide a Description for the configuration.

    4. Provide a Cost Estimate for the configuration. This field is optional.

    5. Click Next

  1. Input Access Control

    1. Add admin, researcher roles to Roles Allowed.

    2. Roles Not Allowed is optional.

    3. Click Next.

  1. Input Parameters to configure the environment.

The Input Parameters tab lists parameters common to all workspace types, plus some parameters specific to the chose instance type.

Explanations and suggested values for common and instance-specific input parameters are provided in the tables below.

Common input parameters

Value Notes
AccessFromCIDRBlock ${cidr} Choose cidr from dropdown list.
EncryptionKeyArn ${encryptionKeyArn} Choose encryptionKeyArn from dropdown list.
EnvironmentInstanceFiles ${environmentInstanceFiles} Choose environmentInstanceFiles from dropdown list.
IamPolicyDocument ${iamPolicyDocument} Choose iamPolicyDocument from dropdown list.
KeyName ${adminKeyPairName} Choose adminKeyPairName from dropdown list.
Namespace ${namespace} Choose namespace from dropdown list.
S3Mounts ${s3Mounts} Choose s3Mounts from dropdown list.
Subnet ${subnetId} Choose subnetId from dropdown list.
VPC ${vpcId} Choose vpcId from dropdown list.

EC2 Linux input parameters

Tip

Information on EC2 instance types is available at Amazon EC2 Instance Types.

Value Notes
AmiId <enter text> The AMI ID (from Amazon Management Console for EC2) for the Linux machine image.
EncryptionKeyArn ${encryptionKeyArn} Choose encryptionKeyArn from dropdown list.
InstanceType <enter text> An EC2 instance type, e.g. t3.small.

EC2 Windows input parameters

Value Notes
AmiId <enter text> The AMI ID (from Amazon Management Console for EC2) for the Windows machine image.
DownloadInterval <enter text> An interval in seconds to wait between two downloads in case of recurring downloads.
InstanceType <enter text> An EC2 instance type, e.g. t3.small.
KeyName ${adminKeyPairName} Choose adminKeyPairName from dropdown list.
Namespace ${namespace} Choose namespace from dropdown list.
RaidDataVolumeSize <enter number> Size of the instance EBS volume.
RecurringDownloads <enter text> true or false
StopRecurringDownloadsAfter -1 Duration after which to stop downloads. Enter -1 to never stop recurring downloads.

SageMaker Input Parameters

Tip

Information on Amazon EC2 instance types that can be used with SageMaker is available at Available SageMaker Studio Instance Types.

Value Notes
AutoStopIdleTimeInMinutes <enter text> Number of idle minutes before auto stop of the instance. 0 disables auto stop.

Amazon EMR Parameters

Tip

For EMR cluster sizing guidelines, see Amazon EMR Cluster Configuration Guidelines and Best Practices.

Value Notes
DiskSizeGB <enter number> EBS Volume size (GB) for each node; provide a value ≥10.
CoreNodeCount <enter number> Number of core nodes to provision (1-80)
MasterInstanceType <enter text> EMR master node EC2 instance type, e.g. m5.xlarge
Market <enter text> Which market to purchase workers on - ON_DEMAND or SPOT.
KeyName <enter text> SSH key pair to use for EMR node login
WorkerBidPrice <enter number> Bid price for the worker spot nodes. This is only applicable when Market = SPOT. Specify 0 for Market = ON_DEMAND.
WorkerInstanceType <enter text> EMR node ec2 instance type, e.g. ml.c4.xlarge.
AmiId <enter text> The AMI ID (from Amazon Management Console for EC2) for the EMR machine image.
  1. After providing input parameters, click Next

  2. Input Tags. This step is optional.

  3. Click Add

  4. The Workspace Configurations tab is displayed showing the new configuration. Additional configurations can be added later.

  5. Click Done

  6. The AWS Service Catalog Products page is displayed. The new workspace type will have the status Not Approved. See Figure 7.

Figure 7: Pending workspace

  1. Click Test launch

  2. Provide a Name and Description, select a Project, and select the new Configuration.

  3. Provide Restricted CIDR. Only IP addresses from within the specified CIDR block can access the Workspace. The default value corresponds to the computer’s IP address.

Note

CIDR (Classless Inter-Domain Routing) is a method for allocating IP addresses and for IP routing. For more information, see Working with VPCs and subnets.

  1. Click Launch

  2. The Research Workspaces page will be displayed, with the new workspace in the Pending state. See Figure 8.

Figure 8: Research Workspaces page showing pending workspace

Note

Launching new workspaces can take five minutes or longer, depending on the resources.

  1. When the Workspace status changes to Available, click Connect. The method used by Service Workbench on AWS to connect to the Workspace depends on the type of compute resources in the Workspace.

For Linux workspaces, Service Workbench on AWS uses SSH through Amazon EC2 Instance Connect. Copy and paste the suggested SSH command line into a terminal window.

For Windows workspaces, Service Workbench on AWS uses Remote Desktop Protocol (RDP).

For Amazon SageMaker, Service Workbench on AWS connects to the Jupyter notebook web interface.

For Amazon EMR, Service Workbench on AWS connects to the Jupyter notebook web interface. Log in with the default password, go-research-on-aws.

  1. After connecting to the workspace correctly, click Workspace Types on the left navigation sidebar.

Note

If the workspace fails to launch, edit the configuration by returning to the Workspace Types page, editing the Workspace Type, then editing the Configuration.

  1. If desired, create additional configurations for the workspace type by repeating the steps above.

  2. Click Approve for the new workspace type. This makes the workspace type available to other users.

  3. Repeat the steps in this section to import additional Service Catalog Products, if appropriate.

Create a research study

Service Workbench on AWS enables organizations to provide researchers with a centralized location to search for studies (data sets) and deploy research workspaces connected to them. Service Workbench on AWS supports three types of studies, described below.

  • My Studies: Studies that are only available to the user that created them. A user can use this to work on datasets that are exclusive to them or that are used specifically for their research.

  • Organization Studies: Studies that have been shared with the Organization. These could be data that had been collected by efforts of the organization or are licensed to the organization. It is possible to grant or deny users access to this data in order to comply with regulations or licensing restrictions on the data.

  • Open Data: Publicly available studies published to Open Data on AWS.

    To create a study in the Service Workbench on AWS web portal:

  1. Click Studies in the left navigation pane.

  1. Click Create Study.

  2. Provide an ID for the Study.

  3. Choose My Study or Organizational Study.

  4. Enter a name for the Study in the Name field.

  5. Enter a description for the Study in the Description field.

  6. Select the Project that this Study relates to in the Project ID drop down field.

  7. Click the Create Study button.