Menu
Amazon Elastic Compute Cloud
User Guide for Windows Instances

Tutorial: Setting Up a Windows HPC Cluster on Amazon EC2

You can launch a scalable Windows High Performance Computing (HPC) cluster using Amazon EC2 instances. A Windows HPC cluster requires an Active Directory domain controller, a DNS server, a head node, and one or more compute nodes.

To set up a Windows HPC cluster on Amazon EC2, complete the following tasks:

For more information about high performance computing, see High Performance Computing (HPC) on AWS.

Prerequisites

  • Install the AWS Command Line Interface tools, and set the region you'll be using as the default region. For more information, see Installing the AWS Command Line Interface in the AWS Command Line Interface User Guide.

  • These procedures assume that you have a VPC in which to launch your instances. You can use your default VPC, or configure a nondefault VPC. For more information, see What is Amazon VPC? in the Amazon VPC User Guide.

Step 1: Set Up Your Active Directory Domain Controller

The Active Directory domain controller provides authentication and centralized resource management of the HPC environment and is required for the installation. To set up your Active Directory, complete these steps:

  1. Create the security groups required for Active Directory.

  2. Create the instance that serves as the domain controller for your HPC cluster.

  3. Configure the domain controller for your HPC cluster.

Creating Security Groups for Active Directory

Use the AWS CLI to create security groups for the domain controller and domain members.

To create the required security groups for Active Directory

  1. Create a security group in your VPC for the domain controller. In the output, take note of the security group ID.

    Copy
    aws ec2 create-security-group --vpc-id vpc-id --group-name "SG - Domain Controller" --description "Active Directory Domain Controller" { "GroupId": "dc-security-group-id" }
  2. Create a security group in your VPC for the domain members. In the output, take note of the security group ID.

    Copy
    aws ec2 create-security-group --vpc-id vpc-id --group-name "SG - Domain Member" --description "Active Directory Domain Member" { "GroupId": "dm-security-group-id" }
  3. Copy the contents of the first file in IP Permissions for the Active Directory Security Groups to a text editor. Replace the dm-security-group-id values with the ID of the your domain member security group. Save the file, using the file name dc-sg-rules.json.

  4. Add the rules to the domain controller security group.

    Copy
    aws ec2 authorize-security-group-ingress --group-id dc-security-group-id --ip-permissions file://dc-sg-rules.json

    Note

    If the JSON file is located in a different directory from which you're working, you must include the path to the file after file://.

  5. Copy the contents of the second file in IP Permissions for the Active Directory Security Groups to a text editor. Replace the dc-security-group-id values with the ID of the your domain controller security group. Save the file, using the file name dm-sg-rules.json.

  6. Add the rules to the domain member security group.

    Copy
    aws ec2 authorize-security-group-ingress --group-id dm-security-group-id --ip-permissions file://dm-sg-rules.json
  7. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

  8. In the navigation pance, choose Security Groups. Verify that the following security groups appear in the list, and are populated with the required rules:

    • SG - Domain Controller

    • SG - Domain Member

Alternatively, manually set up the firewall to allow traffic on the required ports. For more information, go to How to configure a firewall for domains and trusts on the Microsoft website.

Creating the Domain Controller for your HPC cluster

Launch an instance that will serve as the domain controller for your HPC cluster.

To create a domain controller for your HPC cluster

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

    Choose the same region in which you created your security groups.

  2. On the console dashboard, choose Launch Instance.

  3. On the Choose an AMI page, select an AMI for Windows Server, and choose Select.

  4. On the next page of the wizard, select an instance type, then choose Next: Configure Instance Details.

  5. On the Configure Instance Details page, select your VPC from Network and a subnet from Subnet. On the next page of the wizard, you can specify additional storage for your instance.

  6. On the Add Tags page, enter Domain Controller as the value for the Name tag for the instance, and then choose Next: Configure Security Group.

  7. On the Configure Security Group page, choose Select an existing security group, select SG - Domain Controller from the list of security groups, and then choose Review and Launch.

  8. Choose Launch.

After you've launched your instance, associate an Elastic IP with the instance.

To associate an Elastic IP address with an instance

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

  2. In the navigation pane, choose Elastic IPs.

  3. Choose Allocate new address.

  4. When prompted, choose Allocate, and then close the confirmation dialog box.

    Note

    If your account supports EC2-Classic, first choose VPC from the list.

  5. Select the Elastic IP address you created, choose Actions, and then choose Associate address.

  6. In the Instance list, select the Domain Controller instance and then choose Associate.

Configuring the Domain Controller for Your HPC Cluster

Connect to the instance you created, and configure the server as a domain controller for the HPC cluster.

To configure your instance as a domain controller

  1. Connect to your Domain Controller instance. For more information, see Connecting to Your Windows Instance.

  2. Open Server Manager, and add the Active Directory Domain Services role.

  3. Promote the server to a domain controller using Server Manager or by running DCPromo.exe.

  4. Create a new domain in a new forest.

  5. Enter hpc.local as the fully qualified domain name (FQDN).

  6. Select Forest Functional Level as Windows Server 2008 R2.

  7. Ensure that the DNS Server option is selected, and then choose Next.

  8. Select Yes, the computer will use an IP address automatically assigned by a DHCP server (not recommended).

  9. In the warning box, choose Yes to continue.

  10. Complete the wizard and then select Reboot on Completion.

  11. Connect to the instance as hpc.local\administrator.

  12. Create a domain user hpc.local\hpcuser.

Step 2: Configure Your Head Node

An HPC client connects to the head node. The head node facilitates the scheduled jobs. You configure your head node by completing the following steps:

  1. Create security groups for your HPC cluster.

  2. Launch an instance for your head node.

  3. Install the HPC Pack.

  4. Configure your HPC cluster.

Creating Security Groups for Your HPC Cluster

Use the AWS CLI to create a security group for the HPC cluster.

To create the security group for your HPC cluster

  1. Create a security group in your VPC for the HPC cluster. In the output, take note of the security group ID.

    Copy
    aws ec2 create-security-group --vpc-id vpc-id --group-name "SG - Windows HPC Cluster" --description "Windows HPC Server 2008 R2 Cluster Nodes" { "GroupId": "hpc-security-group-id" }
  2. Copy the contents of the JSON file in IP Permissions for HPC Cluster Security Group to a text editor. Replace the hpc-security-group-id value with the ID of your HPC security group. Save the file, using the file name hpc-sg-rules.json.

  3. Add the rules to your HPC cluster security group.

    Copy
    aws ec2 authorize-security-group-ingress --group-id hpc-security-group-id --ip-permissions file://hpc-sg-rules.json
  4. Open the Amazon EC2 console, select Security Groups from the navigation pane, and verify that the SG - Windows HPC Cluster security group appears in the list, and is populated with the required security group rules.

Alternatively, manually configure the firewall with the port requirements for HPC cluster members to communicate. For more information, see Windows Firewall configuration on the Microsoft website.

Launch an Instance for the HPC Head Node

Launch an instance and then configure it as a member of the hpc.local domain and with the necessary user accounts.

To configure an instance as your head node

  1. Launch an instance and name it HPC-Head. When you launch the instance, select both of these security groups:

    • SG - Windows HPC Cluster

    • SG - Domain Member

  2. Connect to the instance and get the existing DNS server address from HPC-Head using the following command:

    Copy
    C:\> IPConfig /all
  3. Update the TCP/IPv4 properties of the HPC-Head NIC to include the Elastic IP address for the Domain Controller instance as the primary DNS, and then add the additional DNS IP address from the previous step.

  4. Join the machine to the hpc.local domain using the credentials for hpc.local\administrator (the domain administrator account).

  5. Add hpc.local\hpcuser as the local administrator. When prompted for credentials, use hpc.local\administrator, and then restart the instance.

  6. Connect to HPC-Head as hpc.local\hpcuser.

Install the HPC Pack

To install the HPC Pack

  1. Connect to your HPC-Head instance using the hpc.local\hpcuser account.

  2. Using Server Manager, turn off Internet Explorer Enhanced Security Configuration (IE ESC) for Administrators.

    1. In Server Manager, under Security Information, choose Configure IE ESC.

    2. Turn off IE ESC for administrators.

  3. Install the HPC Pack on HPC-Head.

    1. Download the HPC Pack to HPC-Head from the Microsoft Download Center. Choose the HPC Pack for the version of Windows Server on HPC-Head.

    2. Extract the files to a folder, open the folder, and double-click setup.exe.

    3. On the Installation page, select Create a new HPC cluster by creating a head node, and then choose Next.

    4. Accept the default settings to install all the databases on the Head Node, and then choose Next.

    5. Complete the wizard.

Configure Your HPC Cluster on the Head Node

To configure your HPC cluster on the head node

  1. Start HPC Cluster Manager.

  2. In the Deployment To-Do List, select Configure your network.

    1. In the wizard, select the default option (5), and then choose Next.

    2. Complete the wizard accepting default values on all screens, and choose how you want to update the server and participate in customer feedback.

    3. Choose Configure.

  3. Select Provide Network Credentials, then supply the hpc.local\hpcuser credentials.

  4. Select Configure the naming of new nodes, and then choose OK.

  5. Select Create a node template.

    1. Select the Compute node template, and then choose Next.

    2. Select Without operating system, and then continue with the defaults.

    3. Choose Create.

Step 3: Set Up the Compute Node

Setting up the compute node involves the following steps:

  1. Launch an instance for your compute node.

  2. Install the HPC Pack on the instance.

  3. Add the compute node to your cluster.

Launch an Instance for the HPC Compute Node

Configure your compute node by launching an instance, and then configuring the instance as a member of the hpc.local domain with the necessary user accounts.

To configure an instance for your compute node

  1. Launch an instance and name it HPC-Compute. When you launch the instance, select the following security groups: SG - Windows HPC Cluster and SG - Domain Member.

  2. Log in to the instance and get the existing DNS server address from HPC-Compute using the following command:

    Copy
    C:\> IPConfig /all
  3. Update the TCP/IPv4 properties of the HPC-Compute NIC to include the Elastic IP address of the Domain Controller instance as the primary DNS. Then add the additional DNS IP address from the previous step.

  4. Join the machine to the hpc.local domain using the credentials for hpc.local\administrator (the domain administrator account).

  5. Add hpc.local\hpcuser as the local administrator. When prompted for credentials, use hpc.local\administrator, and then restart.

  6. Connect to HPC-Compute as hpc.local\hpcuser.

Install the HPC Pack on the Compute Node

To install the HPC Pack on the compute node

  1. Connect to your HPC-Compute instance using the hpc.local\hpcuser account.

  2. Using Server Manager, turn off Internet Explorer Enhanced Security Configuration (IE ESC) for Administrators.

    1. In Server Manager, under Security Information, choose Configure IE ESC.

    2. Turn off IE ESC for administrators.

  3. Install the HPC Pack on HPC-Compute.

    1. Download the HPC Pack to HPC-Compute from the Microsoft Download Center. Choose the HPC Pack for the version of Windows Server on HPC-Compute.

    2. Extract the files to a folder, open the folder, and double-click setup.exe.

    3. On the Installation page, select Join an existing HPC cluster by creating a new compute node, and then choose Next.

    4. Specify the fully-qualified name of the HPC-Head instance, and then choose the defaults.

    5. Complete the wizard.

Add the Compute Node to Your HPC Cluster

To complete your cluster configuration, from the head node, add the compute node to your cluster.

To add the compute node to your cluster

  1. Connect to the HPC-Head instance as hpc.local\hpcuser.

  2. Open HPC Cluster Manager.

  3. Select Node Management.

  4. If the compute node displays in the Unapproved bucket, right-click the node that is listed and select Add Node.

    1. Select Add compute nodes or broker nodes that have already been configured.

    2. Select the check box next to the node and choose Add.

  5. Right-click the node and choose Bring Online.

Step 4: Scale Your HPC Compute Nodes (Optional)

To scale your compute nodes

  1. Connect to the HPC-Compute instance as hpc.local\hpcuser.

  2. Delete any files you downloaded locally from the HP Pack installation package. (You have already run setup and created these files on your image so they do not need to be cloned for an AMI.)

  3. From C:\Program Files\Amazon\Ec2ConfigService open the file sysprep2008.xml.

  4. At the bottom of <settings pass="specialize">, add the following section. Make sure to replace hpc.local, password, and hpcuser to match your environment.

    Copy
    <component name="Microsoft-Windows-UnattendedJoin" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <Identification> <UnsecureJoin>false</UnsecureJoin> <Credentials> <Domain>hpc.local</Domain> <Password>password</Password> <Username>hpcuser</Username> </Credentials> <JoinDomain>hpc.local</JoinDomain> </Identification> </component>
  5. Save sysprep2008.xml.

  6. Choose Start, All Programs, EC2ConfigService Settings.

    1. Choose the General tab, and clear the Set Computer Name check box.

    2. Choose the Bundle tab, and then choose Run Sysprep and Shutdown Now.

  7. Open the Amazon EC2 console.

  8. In the navigation pane, choose Instances.

  9. Wait for the instance status to show stopped.

  10. Select the instance, choose Actions, Image, Create Image.

  11. Specify an image name and image description, and then choose Create Image to create an AMI from the instance.

  12. Start the original HPC-Compute instance that was shut down.

  13. Connect to the head node using the hpc.local\hpcuser account.

  14. From HPC Cluster Manager, delete the old node that now appears in an error state.

  15. In the Amazon EC2 console, in the navigation pane, choose AMIs.

  16. Use the AMI you created to add additional nodes to the cluster.

You can launch additional compute nodes from the AMI that you created. These nodes are automatically joined to the domain, but you must add them to the cluster as already configured nodes in HPC Cluster Manager using the head node and then bring them online.

Running the Lizard Performance Measurement Application

You can optionally run the Lizard application, which measures the computational performance and efficiency that can be achieved by your HPC cluster. Go to http://www.microsoft.com/download/en/details.aspx?id=8433, download the lizard_x64.msi installer, and run the installer directly on your head node as hpc.local\hpcuser.

IP Permissions for the Active Directory Security Groups

The following JSON contains the IP permissions structures for the security groups for your Active Directory environment: one group for Active Directory domain controllers and one for Active Directory domain member servers.

For more information about these security group rules, go to the following Microsoft article: http://support.microsoft.com/kb/179442.

1. Security group rules for the domain controller security group

The following rules apply to the domain controller security group. Replace the dm-security-group-id value with the ID of your domain member security group. Replace the cidr_block value with the CIDR block of your local network.

Copy
[ { "IpProtocol": "UDP", "FromPort": 123, "ToPort": 123, "UserIdGroupPairs": [ { "GroupId": "dm-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 135, "ToPort": 135, "UserIdGroupPairs": [ { "GroupId": "dm-security-group-id" } ] }, { "IpProtocol": "UDP", "FromPort": 138, "ToPort": 138, "UserIdGroupPairs": [ { "GroupId": "dm-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 49152, "ToPort": 65535, "UserIdGroupPairs": [ { "GroupId": "dm-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 389, "ToPort": 389, "UserIdGroupPairs": [ { "GroupId": "dm-security-group-id" } ] }, { "IpProtocol": "UDP", "FromPort": 389, "ToPort": 389, "UserIdGroupPairs": [ { "GroupId": "dm-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 636, "ToPort": 636, "UserIdGroupPairs": [ { "GroupId": "dm-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 3268, "ToPort": 3269, "UserIdGroupPairs": [ { "GroupId": "dm-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 53, "ToPort": 53, "UserIdGroupPairs": [ { "GroupId": "dm-security-group-id" } ] }, { "IpProtocol": "UDP", "FromPort": 53, "ToPort": 53, "UserIdGroupPairs": [ { "GroupId": "dm-security-group-id" } ] }, { "IpProtocol": "UDP", "FromPort": 88, "ToPort": 88, "UserIdGroupPairs": [ { "GroupId": "dm-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 88, "ToPort": 88, "UserIdGroupPairs": [ { "GroupId": "dm-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 445, "ToPort": 445, "UserIdGroupPairs": [ { "GroupId": "dm-security-group-id" } ] }, { "IpProtocol": "UDP", "FromPort": 445, "ToPort": 445, "UserIdGroupPairs": [ { "GroupId": "dm-security-group-id" } ] }, { "IpProtocol": "ICMP", "FromPort": -1, "ToPort": -1, "UserIdGroupPairs": [ { "GroupId": "dm-security-group-id" } ] }, { "IpProtocol": "UDP", "FromPort": 53, "ToPort": 53, "IpRanges": [ { "CidrIp": "cidr_block" } ] }, { "IpProtocol": "TCP", "FromPort": 3389, "ToPort": 3389, "IpRanges": [ { "CidrIp": "cidr_block" } ] } ]

2. Security group rules for the domain member security group

The following rules apply to the domain member security group. Replace the dc-security-group-id value with the ID of your domain controller security group.

Copy
[ { "IpProtocol": "TCP", "FromPort": 49152, "ToPort": 65535, "UserIdGroupPairs": [ { "GroupId": "dc-security-group-id" } ] }, { "IpProtocol": "UDP", "FromPort": 49152, "ToPort": 65535, "UserIdGroupPairs": [ { "GroupId": "dc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 53, "ToPort": 53, "UserIdGroupPairs": [ { "GroupId": "dc-security-group-id" } ] }, { "IpProtocol": "UDP", "FromPort": 53, "ToPort": 53, "UserIdGroupPairs": [ { "GroupId": "dc-security-group-id" } ] } ]

IP Permissions for HPC Cluster Security Group

The following JSON file contains the IP permissions to create a security group for your HPC cluster nodes. Replace the hpc-security-group-id value with the ID of the SG - Windows HPC Cluster security group. The last rule enables you to connect to your instance via RDP. Replace the cidr_block value with the CIDR block for your network.

For more information about these security group rules, go to the following Microsoft article: http://technet.microsoft.com/en-us/library/ff919486.aspx#BKMK_Firewall

Copy
[ { "IpProtocol": "TCP", "FromPort": 80, "ToPort": 80, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 443, "ToPort": 443, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 1856, "ToPort": 1856, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 5800, "ToPort": 5801, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 5801, "ToPort": 5801, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 5969, "ToPort": 5969, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 5970, "ToPort": 5970, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 5974, "ToPort": 5974, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 5999, "ToPort": 5999, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 6729, "ToPort": 6730, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 7997, "ToPort": 7997, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 8677, "ToPort": 8677, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 9087, "ToPort": 9087, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 9090, "ToPort": 9092, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 9100, "ToPort": 9163, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 9200, "ToPort": 9263, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 9794, "ToPort": 9794, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 9892, "ToPort": 9893, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "UDP", "FromPort": 9893, "ToPort": 9893, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 6498, "ToPort": 6498, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 7998, "ToPort": 7998, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 8050, "ToPort": 8050, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 5051, "ToPort": 5051, "UserIdGroupPairs": [ { "GroupId": "hpc-security-group-id" } ] }, { "IpProtocol": "TCP", "FromPort": 3389, "ToPort": 3389, "IpRanges": [ { "CidrIp": "cidr_block" } ] } ]