Install and configure NVMe on the Linux host Configure NVMe on the FSx for ONTAP file system Mount an NVMe device on your Linux client

Provisioning NVMe for Linux

FSx for ONTAP supports the Non-Volatile Memory Express over TCP (NVMe/TCP) block storage protocol. With NVMe/TCP, you use the ONTAP CLI to provision namespaces and subsystems and then map the namespaces to subsystems, similar to the way LUNs are provisioned and mapped to initiator groups (igroups) for iSCSI. The NVMe/TCP protocol is available on second-generation file systems that have 6 or fewer high-availability (HA) pairs..

The examples presented in these procedures demonstrate how to set up NVMe/TCP block storage on your FSx for ONTAP file system and the Linux client, and use the following setup:

Install and configure the NVMe client on the Linux host.
Configure NVMe on the file system's SVM.
- Create an NVMe namespace.
- Create an NVMe subsystem.
- Map the namespace to the subsystem.
- Add the client NQN to the subsystem.
Mount an NVMe device on the Linux client.

You have already created an FSx for ONTAP file system. For more information, see Creating an iSCSI LUN.
Create an EC2 instance running Red Hat Enterprise Linux (RHEL) 9.3, which will be the Linux host on which you will configure NVMe.
The Linux host has VPC security groups configured to allow inbound and outbound traffic as described in File System Access Control with Amazon VPC.
You know the credentials for the ONTAP user with fsxadmin privileges that you will use to access the ONTAP CLI.
The Linux host that you will configure for NVMe and use to access the FSx for ONTAP file system are located in the same VPC and AWS account.

Beyond the scope of the examples presented in this topic if the host is located in another VPC, you can use VPC peering or AWS Transit Gateway to grant other VPCs access to the volume's iSCSI endpoints. For more information, see Accessing data from outside the deployment VPC.

If you are using an EC2 instance running a different Linux AMI than the one used in these procedures and examples, some of the utilities that get installed on the host might already be installed, and you might use different commands to install required packages. Aside from installing packages, the commands used in this section are valid for other EC2 Linux AMIs.
We recommend that the EC2 instance be in the same availability zone as your file system's preferred subnet.

Topics

Install and configure NVMe on the Linux host
Configure NVMe on the FSx for ONTAP file system
Mount an NVMe device on your Linux client

Install and configure NVMe on the Linux host

To install the NVMe client

Connect to your Linux instance using an SSH client. For more information, see Connect to your Linux instance from Linux or macOS using SSH.
Install nvme-cli using the following command:
```
~$ sudo yum install -y nvme-cli
```
Load the nvme-tcp module onto the host:
```
$ sudo modprobe nvme-tcp
```
Get the Linux host's NVMe Qualified Name (NQN) by using the following command:
```
$ cat /etc/nvme/hostnqn
nqn.2014-08.org.nvmexpress:uuid:9ed5b327-b9fc-4cf5-97b3-1b5d986345d1
```
Record the response for use in a later step.

Configure NVMe on the FSx for ONTAP file system

To configure NVMe on the file system

Connect to the NetApp ONTAP CLI on the FSx for ONTAP file system on which you plan to create the NVMe device(s).

To access the NetApp ONTAP CLI, establish an SSH session on the management port of the Amazon FSx for NetApp ONTAP file system by running the following command. Replace management_endpoint_ip with the IP address of the file system's management port.
```
[~]$ ssh fsxadmin@management_endpoint_ip
```
For more information, see Managing file systems with the ONTAP CLI.

Create a new volume on the SVM that you are using to access the NVMe interface.


::> vol create -vserver fsx -volume nvme_vol1 -aggregate aggr1 -size 1t
     [Job 597] Job succeeded: Successful

Create the NVMe namespace rhel using the vserver nvme namespace create NetApp ONTAP CLI command. A namespace maps to initiators (clients) and controls which initiators (clients) have access to NVMe devices.
```
::> vserver nvme namespace create -vserver fsx -path /vol/nvme_vol1/rhel -size 100g -ostype linux
Created a namespace of size 100GB (107374182400).
```
Create the NVMe subsystem using the vserver nvme subsystem create NetApp ONTAP CLI command..
```
~$ vserver nvme subsystem create -vserver fsx -subsystem rhel -ostype linux
```

Map the namespace to the subsystem you just created.


::> vserver nvme subsystem map add -vserver fsx -subsystem rhel -path /vol/nvme_vol1/ns_1

Add the client to the subsystem using its NQN that you retrieved previously.
```
::> vserver nvme subsystem host add -subsystem rhel -host-nqn nqn.2014-08.org.nvmexpress:uuid:ec21b083-1860-d690-1f29-44528e4f4e0e -vserver fsx
```
If you want to make the devices mapped to this subsystem available to multiple hosts, you can specify multiple initiator names separated with a comma. For more information, see vserver nvme subsystem host add in the NetApp ONTAP Docs.

Confirm that the namespace exists using the vserver nvme namespace show command:


::> vserver nvme namespace show -vserver fsx -instance
Vserver Name: fsx
            Namespace Path: /vol/nvme_vol1/ns_1
                      Size: 100GB
                 Size Used: 90.59GB
                   OS Type: linux
                   Comment: 
                Block Size: 4KB
                     State: online
         Space Reservation: false
Space Reservations Honored: false
              Is Read Only: false
             Creation Time: 5/20/2024 17:03:08
            Namespace UUID: c51793c0-8840-4a77-903a-c869186e74e3
                  Vdisk ID: 80d42c6f00000000187cca9
      Restore Inaccessible: false
   Inconsistent Filesystem: false
       Inconsistent Blocks: false
                    NVFail: false
Node Hosting the Namespace: FsxId062e9bb6e05143fcb-01
               Volume Name: nvme_vol1
                Qtree Name: 
          Mapped Subsystem: rhel
            Subsystem UUID: db526ec7-16ca-11ef-a612-d320bd5b74a9               
              Namespace ID: 00000001h
              ANA Group ID: 00000001h
              Vserver UUID: 656d410a-1460-11ef-a612-d320bd5b74a9
                Vserver ID: 3
               Volume MSID: 2161388655
               Volume DSID: 1029
                 Aggregate: aggr1
            Aggregate UUID: cfa8e6ee-145f-11ef-a612-d320bd5b74a9
 Namespace Container State: online
        Autodelete Enabled: false
          Application UUID: -
               Application: -
  Has Metadata Provisioned: true
  
1 entries were displayed.

Use the network interface show -vserver command to retrieve the addresses of the block storage interfaces for the SVM in which you've created your NVMe devices.


::> network interface show -vserver svm_name -data-protocol nvme
            Logical               Status     Network            Current                    Current Is 
Vserver     Interface             Admin/Oper Address/Mask       Node                       Port    Home
----------- ----------            ---------- ------------------ -------------              ------- ----
svm_name
            iscsi_1               up/up      172.31.16.19/20    FSxId0123456789abcdef8-01  e0e     true
            iscsi_2               up/up      172.31.26.134/20   FSxId0123456789abcdef8-02  e0e     true
2 entries were displayed.

Note that the iscsi_1 LIF is used for both iSCSI and NVMe/TCP. In this example, the IP address of iscsi_1 is 172.31.16.19 and iscsi_2 is 172.31.26.134.

Mount an NVMe device on your Linux client

Discover the target iSCSI nodes

On your Linux client, use the following command to discover the target iSCSI nodes, using iscsi_1’s IP address for iscsi_1_IP and the client's IP address for client_IP. Note that iscsi_1 and iscsi_2 LIF are used for both iSCSI and NVMe storage.


~$ sudo nvme discover -t tcp -w client_IP -a iscsi_1_IP


Discovery Log Number of Records 4, Generation counter 11
=====Discovery Log Entry 0======
trtype:  tcp
adrfam:  ipv4
subtype: current discovery subsystem
treq:    not specified
portid:  0
trsvcid: 8009
subnqn:  nqn.1992-08.com.netapp:sn.656d410a146011efa612d320bd5b74a9:discovery
traddr:  172.31.26.134
eflags:  explicit discovery connections, duplicate discovery information
sectype: none
=====Discovery Log Entry 1======
trtype:  tcp
adrfam:  ipv4
subtype: current discovery subsystem
treq:    not specified
portid:  1
trsvcid: 8009
subnqn:  nqn.1992-08.com.netapp:sn.656d410a146011efa612d320bd5b74a9:discovery
traddr:  172.31.16.19
eflags:  explicit discovery connections, duplicate discovery information
sectype: none

(Optional) You can establish additional sessions with the file system. Amazon EC2 has a bandwidth limit of 5 Gb/s (~625 MB/s) for single-flow traffic, but you can create multiple sessions to drive higher levels of throughput to your file system from a single client. For more information, see Amazon EC2 instance network bandwidth in the Amazon Elastic Compute Cloud User Guide for Linux Instances.
Log into the target initiators with a controller loss timeout of at least 1800 seconds, again using iscsi_1’s IP address for iscsi_1_IP and the client's IP address for client_IP. Your NVMe devices are presented as available disks.
```
~$ ~$ sudo nvme connect-all -t tcp -w client_IP -a iscsi_1 -l 1800
```
Use the following command to verify that the NVMe stack has identified and merged the multiple sessions and configured multipathing.
```
~$ cat /sys/module/nvme_core/parameters/multipath
                   Y
```

Verify that the NVMe-oF setting model is set to NetApp ONTAP Controller and load balancing iopolicy is set to round-robin) for the respective ONTAP namespaces correctly reflect on the host.


~$ cat /sys/class/nvme-subsystem/nvme-subsys*/model
Amazon Elastic Block Store              
NetApp ONTAP Controller
~$ cat /sys/class/nvme-subsystem/nvme-subsys*/iopolicy
numa
round-robin

Verify that the namespaces are created and correctly discovered on the host:


~$ sudo nvme list
Node                  Generic               SN                   Model                                    Namespace  Usage                      Format           FW Rev  
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
/dev/nvme0n1          /dev/ng0n1            vol05955547c003f0580 Amazon Elastic Block Store               0x1         25.77  GB /  25.77  GB    512   B +  0 B   1.0     
/dev/nvme2n1          /dev/ng2n1            lWB12JWY/XLKAAAAAAAC NetApp ONTAP Controller                  0x1        107.37  GB / 107.37  GB      4 KiB +  0 B   FFFFFFFF

The new device in the output is /dev/nvme2n1. This naming scheme may differ depending on your Linux installation.

Verify that the controller state of each path is live and has the correct ANA status:


~$ nvme list-subsys /dev/nvme2n1
nvme-subsys2 - NQN=nqn.1992-08.com.netapp:sn.656d410a146011efa612d320bd5b74a9:subsystem.rhel
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:ec2a70bf-3ab2-6cb0-f997-8730057ceb24
               iopolicy=round-robin
\
 +- nvme2 tcp traddr=172.31.26.134,trsvcid=4420,host_traddr=172.31.25.143,src_addr=172.31.25.143 live non-optimized
 +- nvme3 tcp traddr=172.31.16.19,trsvcid=4420,host_traddr=172.31.25.143,src_addr=172.31.25.143 live optimized

Note that the NVMe stack has automatically discovered your file system’s alternate LIF, iscsi_2, 172.31.26.134.

Verify that the NetApp plug-in displays the correct values for each ONTAP namespace device:


~$ sudo nvme netapp ontapdevices -o column
Device           Vserver                   Namespace Path                                     NSID UUID                                   Size     
---------------- ------------------------- -------------------------------------------------- ---- -------------------------------------- ---------
/dev/nvme2n1     fsx                       /vol/nvme_vol1/ns_1                                1    0441c609-3db1-4b0b-aa83-790d0d448ece   107.37GB

To partition the device

Use the following command to verify that the path to your device_name nvme2n1 is present.
```
~$ ls /dev/mapper/nvme2n1
/dev/nvme2n1
```

Partition the disk using fdisk. You’ll enter an interactive prompt. Enter the options in the order shown. Note that the Last sector value will vary depending on the size of your NVMe device (100 GiB in this example). You can make multiple partitions by using a value smaller than the last sector (20971519 in this example).


~$ sudo fdisk /dev/mapper/nvme2n1

The fsdisk interactive prompt starts.


Welcome to fdisk (util-linux 2.37.4). 
Changes will remain in memory only, until you decide to write them. 
Be careful before using the write command. 

Device does not contain a recognized partition table. 
Created a new DOS disklabel with disk identifier 0x66595cb0. 

Command (m for help): n
Partition type 
   p primary (0 primary, 0 extended, 4 free) 
   e extended (container for logical partitions) 
Select (default p): p
Partition number (1-4, default 1): 1 
First sector (256-26214399, default 256): 
Last sector, +sectors or +size{K,M,G,T,P} (256-26214399, default 26214399): 20971519
                                    
Created a new partition 1 of type 'Linux' and of size 100 GiB.
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table. 
Syncing disks.

After entering w, your new partition /dev/nvme2n1 becomes available. The partition_name has the format <device_name><partition_number>. 1 was used as the partition number in the fdisk command in the previous step.

Create your file system using /dev/nvme2n1 as the path.


~$ sudo mkfs.ext4 /dev/nvme2n1

The system responds with the following output:


mke2fs 1.46.5 (30-Dec-2021)
Found a dos partition table in /dev/nvme2n1
Proceed anyway? (y,N) y
Creating filesystem with 26214400 4k blocks and 6553600 inodes
Filesystem UUID: 372fb2fd-ae0e-4e74-ac06-3eb3eabd55fb
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: done

To mount the NVMe device on the Linux client

Create a directory directory_path as the mount point for your file system on the Linux instance.
```
~$ sudo mkdir /directory_path/mount_point
```

Mount the file system using the following command.


~$ sudo mount -t ext4 /dev/nvme2n1 /directory_path/mount_point

(Optional) You can change the ownership of the mount directory to your user. Replace username with your username.
```
~$ sudo chown username:username /directory_path/mount_point
```
(Optional) Verify that you can read from and write data to the file system.
```
~$ echo "Hello world!" > /directory_path/mount_point/HelloWorld.txt
~$ cat directory_path/HelloWorld.txt
Hello world!
```
You have successfully created and mounted an NVMe device on your Linux client.

Warning Javascript is disabled or is unavailable in your browser.

To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

Provisioning iSCSI for Windows

Using FSx for ONTAP with other AWS services