Amazon FSx for Lustre
Lustre User Guide

Getting Started with Amazon FSx for Lustre

Following, you can learn how to get started using Amazon FSx for Lustre. These steps walk you through creating an Amazon FSx for Lustre file system, accessing it from your compute instances, and (optionally) using your Amazon FSx for Lustre file system to process the data in your Amazon S3 bucket with your file-based applications.

This getting started exercise includes these steps:

Prerequisites

To perform this getting started exercise, you need the following:

  • An AWS account with the permissions necessary to create an Amazon FSx for Lustre file system and an Amazon EC2 instance. For more information, see Setting Up.

  • An Amazon EC2 instance running a supported Linux release in your virtual private cloud (VPC) based on the Amazon VPC service. The Lustre client supports Amazon Linux, Amazon Linux 2, CentOS 7.5, RedHat 7.5, SUSE Linux 12 SP3, and Ubuntu 16.04. For this getting started exercise, we recommend CentOS 7.5, which is available in the AWS Marketplace. When creating your Amazon EC2 instance for this getting started exercise, keep the following in mind:

    • We recommend that you create your instance in your default VPC.

    • Verify that an inbound rule exists for the security group you're using with the following values to allow inbound traffic from your Amazon FSx file system.

      • Type: TCP

      • Protocol: 6

      • Port Range: 988

      • Source: Anywhere 0.0.0.0/0

Step 1: Create Your Amazon FSx for Lustre File System

Next, you create your file system in the console.

To create your file system

  1. Open the Amazon FSx for Lustre console.

  2. From the dashboard, choose Create file system to start the file system creation wizard.

  3. Choose FSx for Lustre and then choose Next to display the Create File System page.

  4. Provide information in the File system detail section, following.

    • Provide a name for your file system. You can use up to 256 Unicode letters, white space, and numbers plus the special characters + - = . _ : /.

    • Provide the storage capacity for your file system, in GiB. This value can be any whole number in increments of 3,600 GiB.

    
            File system details section of the Amazon FSx for Lustre Create File System console page.
  5. Provide networking and security group information in the Network & security section, following.

    
            Network & Security section of the Amazon FSx for Lustre Create File System console page.
    • Choose the VPC that you want to associate with your file system. For the purposes of this getting started exercise, choose the same VPC that you chose for your Amazon EC2 instance.

    • Choose any value for the Availability Zones and Subnet.

    • For VPC security groups, the ID for the default security group for your VPC should be already added. If you're not using the default security group, make sure that the following inbound rule is added to the security group you're using for this getting started exercise.

    Type Protocol Port Range Source Description
    TCP 6 988 Custom The ID of this security group Access to your file system over the Lustre protocol
  6. (Optional) For Data repository integration, choose Amazon S3 and specify the Amazon S3 bucket (with optional prefix) as the data repository source.

    Keep Export prefix at the default setting. For more information about the data repository integration, see Using Data Repositories

    Important

    If you link one or more Amazon FSx for Lustre file systems to an Amazon S3 bucket, don't delete the Amazon S3 bucket until all linked file systems have been deleted.

  7. Choose Review and create.

  8. Review the settings for your Amazon FSx for Lustre file system, and choose Create file system.

Now that you've created your file system, make a note of its fully qualified domain name for a later step. You can find the fully qualified domain name for a file system by choosing the name of the file system in the File Systems dashboard, and then choosing Attach.

Step 2: Install and Configure the Lustre Client on your Instance Before Mounting Your File System

To mount your Amazon FSx for Lustre file system from your Amazon EC2 instance, first install the Lustre client.

To download the Lustre client onto your Amazon EC2 instance

  1. Connect to your Amazon EC2 instance. For more information, see Connecting to Your Linux Instance from Windows Using PuTTY or Connecting to Your Linux Instance Using SSH in the Amazon EC2 User Guide for Linux Instances.

  2. Install the Lustre client on your CentOS 7.5 instance with the following procedure:

    1. Open a terminal on your client.

    2. Download and install the Lustre client with the following commands. The client comes in two packages that must be downloaded and installed.

      sudo yum -y install https://downloads.whamcloud.com/public/lustre/lustre-2.10.5/el7.5.1804/client/RPMS/x86_64/kmod-lustre-client-2.10.5-1.el7.x86_64.rpm sudo yum -y install https://downloads.whamcloud.com/public/lustre/lustre-2.10.5/el7.5.1804/client/RPMS/x86_64/lustre-client-2.10.5-1.el7.x86_64.rpm

    Note

    You might need to reboot your compute instance for the client to finish installing.

To mount your file system

  1. Make a directory for the mount point with the following command.

    $ sudo mkdir -p /fsx
  2. Mount the Amazon FSx for Lustre file system to the directory that you created. Use the following command and replace file_system_dns_name with the actual file system's DNS name.

    sudo mount -t lustre -o noatime,flock file_system_dns_name@tcp:/fsx /fsx

    This command mounts your file system using 2 options, -o noatime,flock, following:

    • noatime – mounts your file system with inode access time updates turned off. If you want to update inode access times, use the command without noatime.

    • flock – mounts your file system with file locking enabled. If you do not want file locking enable, use the command without flock.

  3. Verify that the mount command was successful by listing the contents of the directory to which you mounted the file system, /fsx by using the following command.

    $ ls /fsx import-path lustre $

    You can also use the df command, following.

    $ df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1001808 0 1001808 0% /dev tmpfs 1019760 0 1019760 0% /dev/shm tmpfs 1019760 392 1019368 1% /run tmpfs 1019760 0 1019760 0% /sys/fs/cgroup /dev/xvda1 8376300 1263180 7113120 16% / 123.456.789.0@tcp:/fsx 3547698816 13824 3547678848 1% /fsx tmpfs 203956 0 203956 0% /run/user/1000

    The results show the Amazon FSx file system mounted on /fsx.

Step 3: Run Your Analysis

Now that your file system has been created and mounted to a compute instance, you can use it to run your high-performance compute workload.

If you linked your file system to an Amazon S3 data repository, you can export data that you've written to your file system back to your Amazon S3 bucket at any time. From a terminal on one of your compute instances, run the following command to export a file to your Amazon S3 bucket.

sudo lfs hsm_archive filename

For more information on how to run this command on a folder or large collection of files quickly, see Using Data Repositories.

Step 4: Clean Up Resources

After you have finished this exercise, you should follow these steps to clean up your resources and protect your AWS account.

To clean up resources

  1. If you want to do a final export, run the following command.

    nohup find /mnt/fsx -type f -print0 | xargs -0 -n 1 sudo lfs hsm_archive &
  2. On the Amazon EC2 console, terminate your instance. For more information, see Terminate Your Instance in the Amazon EC2 User Guide for Linux Instances.

  3. On the Amazon FSx for Lustre console, delete your file system with the following procedure:

    1. In the navigation pane, choose File systems.

    2. Choose the file system that you want to delete from list of file systems on the dashboard.

    3. For Actions, choose Delete file system.

    4. In the dialog box that appears, confirm that you want to delete your file system, and choose Delete file system.

  4. If you created an Amazon S3 bucket for this exercise, and if you don't want to preserve the data you exported, you can now delete it. For more information, see How Do I Delete an S3 Bucket? in the Amazon Simple Storage Service Console User Guide.