Automated deployment - AWS CloudEndure Migration Factory Solution

Automated deployment

Before you launch the automated deployment, review the architecture, components, and other considerations discussed in this guide. Follow the step-by-step instructions in this section to configure and deploy the AWS CloudEndure Migration Factory Solution into your account.

Time to deploy: Approximately 20 minutes

Prerequisites

Domain permissions

A domain user with local admin permissions to the in-scope source servers targeted for migration is required for Windows and Linux (sudo permissions) servers. If Linux is not in the domain, other users may be used including an LDAP user with sudo permissions or a local sudo user. Before launching this solution, verify that you have the necessary domain permissions or have coordinated with the appropriate person in your organization with domain permissions.

Deployment overview

The procedure for deploying this architecture on AWS consists of the following steps. For detailed instructions, follow the links for each step.

Step 1. Launch the stack

  • Launch the AWS CloudFormation template into your AWS account.

  • Enter values for required parameters: Stack name, Application name, Environment name, Version, Security Group Id, and Subnet Id.

  • Review the template parameters, and adjust if necessary.

Step 2. Create the first user and update the schema

  • Create the initial user and log in to the solution.

  • Update the schema and change the default CloudEndure project name.

Step 3. Build a migration execution server

  • Build a Windows Server 2012 R2 or later server.

  • Install the required packages.

  • Download the sample automation scripts and update the configuration file.

Step 4. Test the solution using the automation scripts

  • Get the necessary domain permissions.

  • Conduct a test run of the migration automation.

Step 5. (Optional) Build a migration tracker dashboard

Step 1. Launch the stack

This automated AWS CloudFormation template deploys the AWS CloudEndure Migration Factory Solution in the AWS Cloud.

Note

You are responsible for the cost of the AWS services used while running this solution. Refer to the Cost section for more details. For full details, refer to the pricing webpage for each AWS service you will be using in this solution.

  1. Sign in to the AWS Management Console and select the button to launch the aws-cloudendure-migration-factory-solution AWS CloudFormation template.

    
                                AWS CloudEndure Migration Factory Solution launch button

    You can also download the template as a starting point for your own implementation.

  2. The template is launched in the US East (N. Virginia) Region by default. To launch this solution in a different AWS Region, use the Region selector in the console navigation bar.

    Note

    This solution uses Amazon Cognito and Amazon QuickSight, which are currently available in specific AWS Regions only. Therefore, you must launch this solution in an AWS Region where these services are available. For the most current availability by Region, refer to the AWS Regional Services List.

  3. On the Create stack page, verify that the correct template URL shows in the Amazon S3 URL text box and choose Next.

  4. On the Specify stack details page, assign a name to your solution stack.

  5. Under Parameters, review the parameters for the template and modify them as necessary. This solution uses the following default values.

    Parameter Default Description
    Application name migration-factory

    Enter a prefix to the AWS CloudFormation Physical ID that identifies the AWS services deployed by this solution.

    Note

    The Application name is used as a prefix to identify the AWS resources that are deployed: <application-name>-<environment-name>-<aws-resource>. If changing the default name, we recommend that you keep the combined prefix labels to 40 characters or less to ensure you do not exceed character limitations.

    Environment name test

    Enter a name to identify the network environment where the solution is deployed. We recommend a descriptive name such as test, dev, or prod.

    Note

    The Environment name is used as a prefix to identify the AWS resources that are deployed: <application-name>-<environment-name>-<aws-resource>. If changing the default name, we recommend that you keep the combined prefix labels to 40 characters or less to ensure you do not exceed character limitations.

    Version 1.4

    The version number of the frontend code that is deployed by Amazon ECS when the solution is initially deployed.

    Tracker true

    By default, the optional migration tracker dashboard is activated, but you can deactivate it by changing this parameter to false.

    Security Group Id default

    Enter the ID that the Amazon ECS cluster will use during the initial deployment of the solution to connect to the endpoint.

    Note

    The default security group ID in your AWS account is used, but you can provide a different ID. The security group must allow outbound access to the internet.

    Subnet Id default Enter the ID that the Amazon ECS cluster will use during the initial deployment of the solution to connect to the endpoint.
    Note

    The default subnet ID in your AWS account is used, but you can provide a different ID. The subnet must allow outbound access to the internet.

    Note

    The solution uses the Security Group Id and Subnet Id from the default Amazon Virtual Private Cloud (Amazon VPC) in your AWS account. If your AWS account does not contain a default VPC, you will need to provide these identifiers from the primary Amazon VPC in your AWS account. For more information, refer to Default VPC and default subnets and Security groups for your VPC in the Amazon Virtual Private Cloud User Guide.

  6. Choose Next.

  7. On the Configure stack options page, choose Next.

  8. On the Review page, review and confirm the settings. Check the box acknowledging that the template will create AWS Identity and Access Management (IAM) resources.

  9. Choose Create stack to deploy the stack.

    You can view the status of the stack in the AWS CloudFormation console in the Status column. You should receive a CREATE_COMPLETE status in approximately 20 minutes.

Step 2. Create the first user and update the schema

Create the initial user and log in to the solution

Use the following procedure to create the initial user.

  1. Navigate to the Amazon Cognito console.

  2. Choose Manage User Pools.

  3. On the Your User Pools page, choose the user pool that starts with the migration-factory prefix.

  4. In the left navigation pane, choose Users and groups.

  5. Select the Users tab and choose Create user.

  6. In the Create user dialog box, Username field, enter an email address and select the checkbox next to Email. Verify that the Send an invitation option is selected.

  7. In the Temporary password field, enter a password.

    Note

    The password must be at least eight characters in length, including upper and lower case letters, numbers, and special characters.

  8. Optionally, enter a Phone Number. If you do not, verify that the Mark phone number as verified option is not checked.

  9. In the Email field, enter the same email address used for the Username. Verify that the Mark email as verified option is selected.

  10. Choose Create user.

    Note

    You will receive an email shortly with the temporary password. Until you change the temporary password, the Account status for this user will display as FORCE_CHANGE_PASSWORD. You can update the password later in the deployment.

Adding a user to the admin group

Use the following procedure to add a user to the default Admin group.

  1. Select the created Username.

  2. On the Users page, select Add to group.

  3. In the dialog box, select the drop-down arrow and choose Admin. Then choose Add to group. This default Admin group authorizes the user to manage migrations in the solution.

Identify the CloudFront URL

Use the following procedure to identify the solution’s Amazon CloudFront URL. This will allow you to log in and change the password.

  1. Navigate to the AWS CloudFormation console and select the solution’s stack.

  2. On the Stacks page, select the Outputs tab and select the Value for the MigrationFactoryURL.

    
                            MigrationFactoryURL value in the AWS CloudFormation Outputs tab

    Figure 3: MigrationFactoryURL value in the AWS CloudFormation Outputs tab

    Note

    If you launched the solution in an AWS Region other than US East (N. Virginia), CloudFront may take longer to deploy and the MigrationFactoryURL may not be accessible immediately (you will receive an access denied error). It might take up to 2 hours before the URL becomes available. The URL includes cloudfront.net as part of the string.

  3. Sign in with your username and temporary password, then create a new password and choose Change Password.

    Note

    The password must be at least eight characters in length, including upper and lower case letters, numbers, and special characters.

Update the schema and change the default CloudEndure project name

  1. On the Migration Factory web interface, select Admin from the drop-down menu on the upper-right corner of the page.

    
                            Migration Factory menu options

    Figure 4: Migration Factory menu options

  2. Select the Attribute Configuration tab.

  3. Select cloudendure_projectname in the left navigation pane.

  4. In the app Attribute Details tab, List Value field, replace the temporary values with a specific name for your CloudEndure project, and choose Save Attribute.

    
                            Migration Factory app Attribute Details tab

    Figure 5: Migration Factory app Attribute Details tab

    Note

    The project name is case sensitive. If you have more than one CloudEndure project, separate the project names with commas.

Step 3. Build a migration execution server

Build a Windows Server 2012 R2 or later server

The migration execution server is used to run migration automation. The server can be created in your on-premises environment or in your AWS Cloud account. To review the server requirements, see Migration Execution Server.

In your migration execution server, use the following procedure to install the packages needed to build the server, download the sample automation scripts, and update the configuration file.

  1. Download Python v3.7.6.

  2. Set the Python installation folder as PATH environment variables. For example: C:\Users\<username>\AppData\Local\Programs\Python\<Python Version>\

  3. Verify that you have administrator privileges, open cmd.exe, and run the following commands to install the Python packages one at a time:

    • python -m pip install requests

    • python -m pip install paramiko

    • python -m pip install boto3

    Note

    If any of these commands fail, upgrade pip by running the following command:

    python -m pip install --upgrade pip

Download the sample automation scripts and update the configuration file

Use the following procedure to download the sample automation scripts and update the configuration file.

  1. Sign in to the migration execution server and download the sample automation scripts from the GitHub repository.

  2. Open cmd.exe and take the following actions:

    1. From the c:\ prompt, create a folder named migrations.

      (c:\migrations)

    2. From the c:\migrations prompt, create a folder named scripts.

      (c:\migrations\scripts)

  3. Unzip the sample automation scripts file and copy all the files to the scripts folder.

  4. Navigate to the AWS CloudFormation console and select the solution’s stack.

  5. Select the Outputs tab and copy the LoginAPI and UserAPI values.

  6. Navigate back to the command window, verify that you are in the c:\migrations\scripts folder, and open the FactoryEndpoints.json file.

    {} FactoryEndpoints.json 1 { 2 "LoginApiUrl":"<loginapi-value>", 3 "UserApiUrl":"<userapi-value>" 4 }

    Replace <loginapi-value> and <userapi-value> with the corresponding values you retrieved from the AWS CloudFormation console. Do not add a slash (/) at the end of the URL.

Step 4. Test the solution using the automation scripts

Access the domains

The sample automation scripts included with this solution connect to the in-scope source servers to automate migration tasks, such as the installation of the CloudEndure agent, and shutting down the source servers. In order to conduct a test run of the solution, a domain user with local admin permissions to the source servers is required, for Windows and Linux (sudo permissions) servers. If Linux is not in the domain, other users such as LDAP user with sudo permissions or a local sudo user may be used. For more information about automated migration tasks, refer to Automated migration activities.

Conduct a test run of the migration automation

This solution lets you to conduct a test run of the migration automation. Using automation scripts, the migration process imports the data from the migration CSV file into the solution. Prerequisite checks are conducted for the source servers, the CloudEndure agent is pushed to the source servers, replication status is verified, and the execution migration server is launched from the Migration Factory web interface. For step-by-step instructions on running a test, refer to Automated migration activities.

Step 5. (Optional) Build a migration tracker dashboard

If you deployed the optional migration tracker component, you can set up a QuickSight dashboard that will visualize the migration metadata stored in the Amazon DynamoDB table. Use the following procedures to (1) set up the QuickSight permissions and connections and (2) create a dashboard.

Note

If the Migration Factory is empty and there is no wave, application, and server data, then there will not be any data to build a QuickSight dashboard.

Set the permission and connections

If you have not set up Amazon QuickSight in your AWS account, refer to Setting Up for Amazon QuickSight in the Amazon QuickSight User Guide. After you have set up a QuickSight subscription, use the following procedure to set the permissions and connections between QuickSight and this solution.

Note

This solution uses Amazon QuickSight enterprise license. However if you don’t want the email reporting, insights, and the hourly data refresh, you can opt for a standard license, which can also be used with migration tracker.

First, connect QuickSight with the Amazon S3 bucket:

  1. Navigate to the QuickSight console.

  2. On the QuickSight page, in the top right corner, choose the icon displaying your username to access the drop-down menu and select Manage QuickSight.

  3. On the Account name page, from the left menu pane, select Security & permissions.

  4. On the Security & permissions page, under the QuickSight access to AWS services section, select Add or remove.

  5. From the QuickSight access to AWS services page, tick the checkbox for Amazon S3.

  6. On the Select Amazon S3 buckets dialog box, verify that you are in the S3 Buckets Linked to QuickSight Account tab and tick both the right and left checkboxes for the athena-results and migration-tracker  S3 buckets, as shown in Figure 6.

    QuickSight select Amazon S3 buckets dialog box

    Figure 6: QuickSight select Amazon S3 buckets dialog box

  7. Choose Finish.

Next, set up permissions for Amazon Athena:

  1. From the QuickSight access to AWS services page, tick the checkbox for Amazon Athena.

  2. On the Amazon Athena permissions dialog box, choose Next.

  3. On the Amazon Athena resources dialog box, verify that you are in the S3 Buckets Linked to QuickSight Account tab and verify that the same S3 buckets are checked - athena-results and migration-tracker, as shown in Figure 7.

    QuickSight Amazon Athena resources dialog box

    Figure 7: QuickSight Amazon Athena resources dialog box

  4. Choose Finish.

  5. From the QuickSight access to AWS services page, choose Update.

Next, set up a new analysis:

  1. On the Analysis page, choose New Analysis.

  2. Choose New dataset.

  3. On the Create a Data Set page, choose Athena.

  4. In the New Athena data source dialog box, take the following actions:

    1. In the Data source name field, enter a name for the data source

    2. In the Athena workgroup field, select the appropriate <migration-factory>-workgroup.

    Note

    If you have deployed this solution multiple times, there will be more than one workgroup. Select the one that was created for your current deployment.

    New Athena data source dialog box

    Figure 8: New Athena data source dialog box

  5. Choose Validate connection to ensure that QuickSight can communicate with Athena.

  6. After the connection is validated, choose Create data source.

  7. In the next dialog box, Choose your table, take the following actions:

    1. From the Catalog list, choose AwsDataCatalog.

    2. From the Database list, choose <Athena-table>-tracker.

    3. From the Tables list, choose <tracker-name>-general-view.

    4. Choose Select.

    Choose your table dialog box

    Figure 9: Choose your table dialog box

  8. In the next dialog box, Finish data set creation, select Visualize.

    Finish data set creation dialog box

    Figure 10: Finish data set creation dialog box

    After the data is imported, you will be redirected to the Analysis page. However, before creating your visuals, set up a schedule to refresh your dataset.

  1. Return to the QuickSight homepage by choosing the QuickSight logo.

  2. From the left menu pane, choose Datasets.

  3. On the Datasets page, select the <migration-factory>-general-view dataset.

    QuickSight Datasets page

    Figure 11: QuickSight Datasets page

  4. In the <migration-factory>-general-view dialog box, choose Schedule refresh.

    Migration tracker general view dialog box

    Figure 12: Migration tracker general view dialog box

  5. Choose Create.

  6. In the Create a schedule dialog box, select the appropriate time zone, the refresh frequency, and enter a starting time.

  7. Choose Create.

    Create a schedule dialog box

    Figure 13: Create a schedule dialog box

Create a dashboard

Amazon QuickSight offers the flexibility of building a custom dashboard that helps you to visualize your migration metadata. The following tutorial creates a dashboard containing a count visual that shows the server count by waves and bar charts showing the migration status, as shown in Figure 14. You can customize this dashboard to meet your business needs.

Example QuickSight dashboard

Figure 14: Example QuickSight dashboard

Use the following steps to create a count overview by migration waves. This view counts all the servers in the dataset that are grouped per wave, providing a granular view of the total number of servers in a wave. To create this view, you will convert the server_name into a measure, which allows you to count distinct servers names. Then you will create a wave by wave filter.

  1. From the QuickSight Visualize page, choose + Add and select Add Visual.

  2. From the left Dataset pane, open the drop-down list and select server_name. Choosing this field ensures you get the unique entries.

  3. Hover over the server_name and choose the ellipsis to the right.

    QuickSight Visualize a data set page

    Figure 15: QuickSight Visualize a data set page

  4. Select Convert to measure to convert the dataset from a dimension to a measure. The server_name text turns green to indicate that the dataset has been converted to a measure.

  5. Select server_name to visualize the image. The visual will contain an error message indicating that the field data types must be updated.

  6. On the Field wells page, Value box, choose the drop-down arrow for server_name (Sum) , select Aggregate: Sum, then select Count distinct.

    Field wells page

    Figure 16: Field wells page

The visual should now show a count of the number of unique server names you have in your dataset. You can resize the visualization as needed to ensure it displays the information clearly on your monitor.

Note

You may need to convert your dataset back to dimension when you create another visual.

Next, add filters to the visualization to identify the server count for each migration wave. The following steps will apply a wave_id filter to your visualization.

  1. Verify that the visualization is selected and in the left menu pane, select Filter.

  2. From the left Filters pane, choose the + button and then, in the drop-down list, select wave_id.

    Filters pane drop-down list

    Figure 17: Filters pane drop-down list

  3. In the Edit filter pane, choose wave_id and then under Filter type, select the checkbox next to the value 1.

  4. In the visualization, change the title to Wave 1 Server Count.

Repeat these steps for the other waves that are visualized in your dashboard.

Edit filter pane

Figure 18: Edit filter pane

The next visualization we will add in the dashboard is a doughnut graph showing servers that are in progress of being migrated versus ones that have completed the migration. This chart uses Super-fast, Parallel, In-memory Calculation Engine (SPICE) Queries by creating a new column in the dataset that determines that an incomplete status will be identified as in progress. All values in the dataset that are not completed are combined and categorized as in progress.

Doughnut graph and bar chart visualizing migration progress

Figure 19: Doughnut graph and bar chart visualizing migration progress

Note

By default, when there is no custom query applied to the dataset, up to five migration/replication statuses can be shown. For this solution, a MigrationStatusSummary query is created in a new column: ifelse(migration_status ='Cutover instance launched', 'Completed', 'InProgress')

This query combines the values of the statuses to create one column that is used for the visualization. For information about creating a query, refer to Using the Query Editor in the Amazon QuickSight User Guide.

Use the following steps to create the MigrationStatusSummary column:

  1. In the left menu pane, select Datasets.

  2. On the Datasets page, select the <migration-factory>-general-view dataset.

  3. In the dataset dialog box, select Edit data set.

    Migration factory dataset dialog box

    Figure 20: Migration factory dataset dialog box

  4. In the next dialog box, in the Query mode pane, choose Add calculated field.

    Query mode dialog box

    Figure 21: Query mode dialog box

  5. In the Add calculated field dialog box, enter a name for your SQL query, for example, MigrationStatusSummary.

  6. Enter the following SQL query into the SQL editor:

    ifelse(migration_status ='Cutover instance launched', 'Completed', 'InProgress')
  7. Choose Save.

    Add calculated field dialog box

    Figure 22: Add calculated field dialog box

  8. In the Data page, choose Save & visualize.

    Data page

    Figure 23: Data page

Your newly added query will be listed in the Data set Fields list.

Data set Fields list

Figure 24: Data set Fields list

Next, you will build the dashboard.

  1. On the Visualize page, choose + Add.

  2. In the Dataset pane, access the drop-down list and select MigrationStatusSummary as the main value (x-axis) and wave_id as the grouping.

    Dataset page

    Figure 25: Dataset page

If you have an enterprise license for Amazon QuickSight, insights will be generated after the custom columns are created. You can customize your narratives for each insight. For example:

Example dashboard insights

Figure 26: Example dashboard insights

You can also customize the data by breaking down the metadata into waves. For example:

Example wave 1 server breakdown

Figure 27: Example wave 1 server breakdown

(Optional) View Insights on the QuickSight dashboard

Note

You can use the following procedure if you have an enterprise license for Amazon QuickSight.

Use the following steps to add an insight to your dashboard which shows a breakdown of completed and in progress migrations.

  1. On the left menu pane, choose Insights.

    Insights available with an enterprise license

    Figure 28: Insights available with an enterprise license

  2. On the Insights page, in the Count of Records BY MIGRATIONSTATUSSUMMARY section, hover over Top 2 MigrationSummarys item and choose the + button to add an insight to the visual.

    Add an insight to a visual

    Figure 29: Add an insight to a visual

  3. Customize the insight for your analysis by choosing Customize Narrative on the visual.

    Add an Insight to your dashboard

    Figure 30: Add an Insight to your dashboard

    Customize narrative option

    Figure 31: Customize narrative option

  4. Edit the narrative to fit your use case and choose Save. For example:

    Edit your narrative

    Figure 32: Edit your narrative

    Return to the dashboard and filter it to show each wave:

  5. On the left menu pane, choose Filter.

  6. Choose the + button and select wave_id.

  7. Select a wave to visualize and choose Apply.

    Edit filter pane

    Figure 33: Edit filter pane

  8. To visualize all the migration waves, duplicate the visuals by choosing the ellipsis to the left side of the visual and selecting Duplicate visual to.

    Visualize the migration waves

    Figure 34: Visualize the migration waves

  9. Modify the filter for each visual to show a breakdown for each migration wave.

This insight is customized summarize the total count of servers across all waves. For more information and guide on how to customize insights, refer to Working with Insights in the QuickSight User Guide. You can access this QuickSight dashboard from any device and seamlessly embed it into your applications, portals, and websites. For more information about QuickSight dashboards, refer to Working with Dashboards in the Amazon QuickSight User Guide.