Sign In to the Console
Amazon Elastic Compute Cloud
User Guide for Linux Instances

AWS Documentation » Amazon EC2 » User Guide for Linux Instances » Networking in Amazon EC2 » Elastic Fabric Adapter » Getting Started with EFA and MPI

Getting Started with EFA and MPI

This tutorial helps you to launch an EFA and MPI-enabled instance cluster for HPC workloads. In this tutorial, you will perform the following steps:

Step 1: Prepare an EFA-Enabled Security Group

An EFA requires a security group that allows all inbound and outbound traffic to and from the security group itself.

To create an EFA-enabled security group

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

  2. In the navigation pane, choose Security Groups and then choose Create Security Group.

  3. In the Create Security Group window, do the following:

    1. For Security group name, enter a descriptive name for the security group, such as EFA-enabled security group.

    2. (Optional) For Description, enter a brief description of the security group.

    3. For VPC, select the VPC into which you intend to launch your EFA-enabled instances.

    4. Choose Create.

  4. Select the security group that you created, and on the Description tab, copy the Group ID.

  5. On the Inbound and Outbound tabs, do the following:

    1. Choose Edit.

    2. For Type, choose All traffic.

    3. For Source, choose Custom.

    4. Paste the security group ID that you copied into the field.

    5. Choose Save.

Step 2: Launch a Temporary Instance

Launch a temporary instance that you can use to install and configure the EFA software components. You use this instance to create an EFA-enabled AMI from which you can launch your EFA-enabled instances.

To launch a temporary instance

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

  2. Choose Launch Instance.

  3. On the Choose an AMI page, choose Select for one of the following supported AMIs: Amazon Linux, Amazon Linux 2, RHEL 7.6, RHEL 7.7, CentOS 7, Ubuntu 16.04, and Ubuntu 18.04.

  4. On the Choose an Instance Type page, select one of the following supported instance types and then choose Next: Configure Instance Details: c5n.18xlarge, c5n.metal, i3en.24xlarge, inf1.24xlarge, m5dn.24xlarge, m5n.24xlarge, r5dn.24xlarge, r5n.24xlarge, and p3dn.24xlarge.

  5. On the Configure Instance Details page, do the following:

    1. For Elastic Fabric Adapter, choose Enable.

    2. In the Network Interfaces section, for device eth0, choose New network interface.

    3. Choose Next: Add Storage.

  6. On the Add Storage page, specify the volumes to attach to the instances in addition to the volumes specified by the AMI (such as the root device volume), and then choose Next: Add Tags.

  7. On the Add Tags page, specify a tag that you can use to identify the temporary instance, and then choose Next: Configure Security Group.

  8. On the Configure Security Group page, for Assign a security group, select Select an existing security group, and then select the security group that you created in Step 1.

  9. On the Review Instance Launch page, review the settings, and then choose Launch to choose a key pair and to launch your instance.

Step 3: Install Libfabric and Open MPI

Install the EFA-enabled kernel, EFA drivers, libfabric, and Open MPI stack that is required to support EFA on your temporary instance.

To install libfabric and Open MPI on your temporary instance

  1. Connect to the instance you launched in Step 2. For more information, see Connect to Your Linux Instance.

  2. Download the EFA software installation files. To download the latest stable version, use the following command. 

    $ curl -O https://s3-us-west-2.amazonaws.com/aws-efa-installer/aws-efa-installer-1.7.1.tar.gz

    You can also get the latest version by replacing the version number with latest in the preceding command.

  3. The software installation files are packaged into a compressed .tar.gz file. Extract the files from the compressed .tar.gz file and navigate into the extracted directory. 

    $ tar -xf aws-efa-installer-1.7.1.tar.gz
    $ cd aws-efa-installer

  4. Run the EFA software installation script.

    $ sudo ./efa_installer.sh -y

    Libfabric is installed in the /opt/amazon/efa directory, while Open MPI is installed in the /opt/amazon/openmpi directory.

  5. Log out of the instance and then log back in.

  6. Confirm that the EFA software components were successfully installed.

    $ fi_info -p efa

    The command should return information about the libfabric EFA interfaces. The following example shows the command output. 

    provider: efa
    fabric: EFA-fe80::94:3dff:fe89:1b70
    domain: efa_0-rdm
    version: 2.0
    type: FI_EP_RDM
    protocol: FI_PROTO_EFA
provider: efa
    fabric: EFA-fe80::94:3dff:fe89:1b70
    domain: efa_0-dgrm
    version: 2.0
    type: FI_EP_DGRAM
    protocol: FI_PROTO_EFA
provider: efa;ofi_rxd
    fabric: EFA-fe80::94:3dff:fe89:1b70
    domain: efa_0-dgrm
    version: 1.0
    type: FI_EP_RDM
    protocol: FI_PROTO_RXD

Step 4: (Optional) Install Intel MPI

Note

If you intend to use Open MPI, skip this step. Perform this step only if you intend to use Intel MPI.

Intel MPI requires an additional installation and environment variable configuration.

Prerequisites

Ensure that the user performing the following steps has sudo permissions.

To install Intel MPI

  1. To download the Intel MPI installation files, see the Intel Developer Zone website.

    You must register before you can download the installation files. After you have registered, do the following:

    1. For Product, choose Intel MPI Library for Linux.

    2. For Version, choose 2019 Update 6.

    3. Choose the button that has the .tar.gz file name. For example, l_mpi_2019.6.154.tgz.

  2. The installation files are packaged into a compressed .tar.gz file. Extract the files from the compressed .tar.gz file and navigate into the extracted directory. 

    $ tar -xf file_name.tgz
    $ cd directory_name

  3. Open silent.cfg using your preferred text editor. In line 10, change ACCEPT_EULA=decline to ACCEPT_EULA=accept. Save the changes and close the file.

  4. Run the installation script.

    $ sudo ./install.sh -s silent.cfg

    Intel MPI is installed in the /opt/intel/impi/ directory by default.

  5. Add the Intel MPI environment variables to the corresponding shell startup scripts to ensure that they are set each time that the instance starts. Do one of the following depending on your shell.

    • For bash, add the following environment variable to /home/username/.bashrc and /home/username/.bash_profile. 

      source /opt/intel/impi/2019.6.154/intel64/bin/mpivars.sh

    • For csh and tcsh, add the following environment variable to /home/username/.cshrc. 

      source /opt/intel/impi/2019.6.154/intel64/bin/mpivarsh.csh

  6. Log out of the instance and then log back in.

  7. Run the following command to confirm that Intel MPI was successfully installed.

    $ which mpicc

    The following example shows the command output.

    /opt/intel/compilers_and_libraries_2020.0.154/linux/mpi/intel64/bin/mpicc

Note

If you no longer want to use Intel MPI, remove the environment variables from the shell startup scripts.

Step 5: Install Your HPC Application

Install the HPC application on the temporary instance. The installation procedure varies depending on the specific HPC application. For more information about installing software on your Linux instance, see Managing Software on Your Linux Instance.

Note

You might need to refer to your HPC application’s documentation for installation instructions.

Step 6: Create an EFA-Enabled AMI

After you have installed the required software components, you create an EFA and NCCL-enabled AMI that you can reuse to launch your EFA and NCCL-enabled instances.

To create an AMI from your temporary instance

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

  2. In the navigation pane, choose Instances.

  3. Select the instance that you created in Step 1 and choose Actions, Image, Create Image.

  4. In the Create Image window, do the following:

    1. For Image name, enter a descriptive name for the AMI.

    2. (Optional) For Image description, enter a brief description of the AMI.

    3. Choose Create Image and then choose Close.

  5. In the navigation pane, choose AMIs.

  6. Locate the AMI you created in the list. Wait for the Status to transition from pending to available before continuing to the next step.

Step 7: Launch EFA-Enabled Instances into a Cluster Placement Group

Launch your EFA-enabled instances into a cluster placement group using the EFA-enabled AMI that you created in Step 6, and the EFA-enabled security group that you created in Step 1.

Note

It is not an absolute requirement to launch your EFA-enabled instances into a cluster placement group. However, we do recommend running your EFA-enabled instances in a cluster placement group as it launches the instances into a low-latency group in a single Availability Zone.

To launch your EFA-enabled instances into a cluster placement group

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

  2. Choose Launch Instance.

  3. On the Choose an AMI page, choose My AMIs, find the AMI that you created in Step 6, and then choose Select.

  4. On the Choose an Instance Type page, select one of the following supported instance types and then choose Next: Configure Instance Details: c5n.18xlarge, c5n.metal, i3en.24xlarge, inf1.24xlarge, m5dn.24xlarge, m5n.24xlarge, r5dn.24xlarge, r5n.24xlarge, and p3dn.24xlarge.

  5. On the Configure Instance Details page, do the following:

    1. For Number of instances, enter the number of EFA-enabled instances that you want to launch.

    2. For Network and Subnet, select the VPC and subnet into which to launch the instances.

    3. For Placement group, select Add instance to placement group.

    4. For Placement group name, select Add to a new placement group, enter a descriptive name for the placement group, and then for Placement group strategy, select cluster.

    5. For EFA, choose Enable.

    6. In the Network Interfaces section, for device eth0, choose New network interface. You can optionally specify a primary IPv4 address and one or more secondary IPv4 addresses. If you're launching the instance into a subnet that has an associated IPv6 CIDR block, you can optionally specify a primary IPv6 address and one or more secondary IPv6 addresses.

    7. Choose Next: Add Storage.

  6. On the Add Storage page, specify the volumes to attach to the instances in addition to the volumes specified by the AMI (such as the root device volume), and then choose Next: Add Tags.

  7. On the Add Tags page, specify tags for the instances, such as a user-friendly name, and then choose Next: Configure Security Group.

  8. On the Configure Security Group page, for Assign a security group, select Select an existing security group, and then select the security group that you created in Step 1.

  9. Choose Review and Launch.

  10. On the Review Instance Launch page, review the settings, and then choose Launch to choose a key pair and to launch your instances.

Step 8 Terminate the Temporary Instance

At this point, you no longer need the temporary instance that you launched in Step 1. You can terminate the instance to stop incurring charges for it.

To terminate the temporary instance

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

  2. In the navigation pane, choose Instances.

  3. Select the temporary instance that you created in Step 1 and then choose Actions, Instance State, Terminate, Yes, Terminate.