Setting up Qumulo in Nimble Studio

Amazon Nimble Studio supports the Amazon FSx server at deployment. Nimble Studio also supports third-party storage providers, such as Qumulo.

Qumulo is a highly scalable, and fault tolerant file server software with Server Message Block (SMB) and Network File System (NFS) interfaces. It is propriety software (not open source) and can be deployed in the cloud as well as on-premises. Qumulo offers a CloudFormation template that you find on GitHub.

This tutorial shows you how to install Qumulo an actively running Nimble Studio.

Prerequisites

• To complete this tutorial, you need an active Nimble Studio cloud studio deployed in your AWS account. If you don’t have a cloud studio already deployed, see the Deploying a new studio with StudioBuilder tutorial.

Step 1: Gather information

You need to gather some information about your Nimble Studio cloud studio resources before you can deploy the Qumulo cluster in Step 4: Deploy Qumulo using CloudFormation. This information will also be used in Step 6: Post deployment tasks.

Find the Nimble Studio VPC CIDR

In the following steps you’ll find the IPv4 Classless Inter-Domain Routing (CIDR) of your studio’s Amazon Virtual Private Cloud (Amazon VPC).

1. Sign in to the v.

2. Go to Services → VPC .

3. Select Your VPCs in the left navigation pane, under Virtual Private Cloud.

4. Choose your studio’s VPC.

   1. Your studio VPC is named <your-studio-name>.

5. In the Details section, notice the IPv4 CIDR. You will use this information in Step 4: Deploy Qumulo using CloudFormation.

Find the Workstation subnet Availability Zone

1. Go to Services → VPC .

2. Select Subnets in the left navigation pane.

3. Choose the subnet named Workstations.

4. In the Details section, notice the Availability Zone. You will use this information in the following section, Find the WorkerSupport subnet ID.

Find the WorkerSupport subnet ID

1. Go to Services → VPC .

2. Select Subnets in the left navigation pane.

3. There are two subnets named WorkerSupport. Do the following for both subnets.

4. Select the subnet.

5. In the Detail section, find the Availability Zone.

6. If the Availability Zone matches the Workstations subnet’s Availability Zone, notice the following information. You will use this information in Step 4: Deploy Qumulo using CloudFormation.

   1. Subnet ID

   2. IPv4 CIDR range of that WorkerSupport subnet

Find the Active Directory information

AWS Directory Service for Microsoft Active Directory is a Microsoft Active Directory hosted on the AWS Cloud. To join Qumulo with AWS Managed Microsoft AD, you must first find the AWS Managed Microsoft AD information using the following steps.

1. Go to Services → Directory Service .

2. Select Directories under Active Directories in the left navigation pane.

3. Select the AWS Managed Microsoft AD that Nimble Studio deployed. That Directory name is <your-studio-name>.nimble.<studio-region>.aws

4. Notice the following information. You will use this information in Step 4: Deploy Qumulo using CloudFormation and in Step 6: Post deployment tasks.

   1. Directory DNS name

   2. Directory NetBIOS name

   3. Directory DNS addresses (both IP addresses)

Step 2: Prepare the network

Before deploying Qumulo, you need to reconfigure your security groups to work with Qumulo.

1. Sign in to the AWS Management Console and open the Amazon VPC console.

2. Select Network ACLs under Security on the left navigation menu.

3. Update the WorkerSupport, Workstations, Active Directory, and RenderWorkers security groups by modifying their inbound and outbound rules.

4. Select the security group that you want to modify.

5. Select Actions → Edit inbound rules if you want to add or modify an inbound rule, or Actions → Edit outbound rules if you want to add or modify an outbound rule.

   

6. The following table describes how each of the security groups’ rules should be modified.

NACL name	Inbound/Outbound	Operation	Rule number	Protocol	Ports	Source	Destination
WorkerSupport	Inbound	MODIFY	10	TCP	0-65535	0.0.0.0
	Inbound	ADD	30000	All Traffic		10.0.0.0/16
	Outbound	ADD	30000	All Traffic			10.0.0.0/16
Workstations	Inbound	ADD	30000	All Traffic		10.0.0.0/16
Active Directory	Inbound	ADD	30000	All Traffic		IPv4 CIDR WorkerSupport subnet
	Outbound	ADD	30000	All TCP			IPv4 CIDR WorkerSupport subnet
	Outbound	ADD	30100	All UDP			IPv4 CIDR WorkerSupport subnet
RenderWorkers	Inbound	ADD	30000	All Traffic		10.0.0.0/16

If you use VPC endpoints

If your studio was deployed with a VPC endpoint instead of a default AWS public endpoint, you’ll need to complete one additional step to configure the network. This is to ensure that your VPC endpoints can be reached from the WorkerSupport subnet.

1. Sign in to the AWS Management Console and open the Amazon VPC console.

2. Select Subnets under the Virtual Private Cloud section.

3. Choose the WorkerSupport subnet.

   1. Notice the IPv4 CIDR, because you will use it in step 4b.

      

4. Select Security Groups under the Security section.

5. Select the Vpc Interface Endpoints security group.

6. Select Actions → Edit Inbound Rules.

7. Select Add Rule.

   1. Set the type to HTTPS.

   2. Set the Source to the IPv4 CIDR range of the WorkerSupport subnet.

8. Select Save rules.

Step 3: Prepare Qumulo deployment

To deploy Qumulo, you must pull the latest version of Qumulo from GitHub and upload it into a new Amazon Simple Storage Service (Amazon S3) bucket.

To pull the latest version from the Github repo

1. Go to the aws-sa-waf-cluster GitHub repo.

2. There are two ways to download the aws-sa-waf-cluster code from GitHub:

   1. On the GitHub website, select Code → Download ZIP and unzip the downloaded file.

      

   2. In the Windows command line (you must have git installed) or in the Linux terminal, run the following command: git clone https://github.com/Qumulo/aws-sa-waf-cluster.git

3. Go to the folder named aws-sa-waf-cluster.

To upload to a new S3 bucket

1. Go to Services → S3 .

2. In the AWS Region selector (top-right navigation bar), make sure that the Region for your studio is selected.

   1. Notice this Region. This will be used in Step 4: Deploy Qumulo using Cloud Formation.

3. Select Create bucket.

4. Give your bucket a unique name. For example: qumulo-deployment-<your account id>

   1. Notice the bucket name. This will be used in Step 4: Deploy Qumulo using Cloud Formation.

5. Leave the other sections as their default values and choose Create bucket.

6. After the bucket is created, open it.

7. Upload the aws-sa-waf-cluster folder to the bucket.

8. In the S3 console, select aws-sa-waf-cluster.

9. Select the file named qcluster-existingVPC.cft.yaml.

10. Select Copy URL. This will be used in Step 4: Deploy Qumulo using Cloud Formation.

    

Step 4: Deploy Qumulo using CloudFormation

Qumulo frequently changes their deployment template—and the parameters, with it. However, there are a few key elements that remain unchanged. This section shows which settings need to be configured during deployment so that Qumulo can work with Nimble Studio. However, it doesn’ts include comprehensive instructions about filling out every parameter.

1. Sign in to the AWS Management Console and open the AWS CloudFormation console.

2. Select Create stack → With new resources (standard).

3. In the Prepare template section, choose Template is ready.

4. In the Specify template section, choose Amazon S3 URL as the Template source.

5. Paste the Object URL that you found in To upload to a new S3 bucket.

   

6. Select Next.

   Note

   Qumulo changes their deployment template and parameters quite often. The key elements in the following steps will remain the same and are needed to configure Qumulo to work with Nimble Studio.

7. In the Stack name section, give your stack a name. For example: <your-studio-name>qumulo

8. In the Parameters section, enter the following values:

   1. S3 Bucket Name: Enter the name of the S3 bucket you created in To upload to a new S3 bucket.

   2. S3 Bucket Region: Enter the region of the bucket you created in To upload to a new S3 bucket.

   3. AWS Key-Pair Name: Choose the Key-Pair to be used; you can create a new one, or choose an existing key pair

   4. Amazon VPC ID: Select the Nimble Studio VPC named <your-studio-name>

   5. Qumulo Security Group CIDR #1: Enter the Nimble VPC IPv4 CIDR range that you found in Find the Nimble Studio VPC CIDR.

   6. AWS Private Subnet ID: Select the WorkerSupport subnet that you found in Find the WorkerSupport subnet ID.

   7. Qumulo Sidecar Lambdas Private Subnet ID: Select the WorkerSupport subnet that you found in Find the WorkerSupport subnet ID.

   8. AWS Public Subnet ID: Select the WorkerSupport subnet that you found in Find the WorkerSupport subnet ID.

   9. OPTIONAL: FQDN for R53 Private Hosted Zone: qumulo.local

   10. Qumulo Cluster Admin Password: Enter a secure password. You will use this password to log in to the Qumulo web dashboard in Step 6: Post deployment tasks.

       1. CloudFormation recommends using parameters for sensitive data. For more information about using parameters with CloudFormation, see Security best practices for AWS CloudFormation in the AWS CloudFormation User Guide.

9. Select Next to go to the Configure stack options section.

10. Select Next to go to the Review section.

11. Check the boxes to acknowledge that CloudFormation might automatically create and modify your resources.

    

12. Select Create stack.

Watch the deployment to ensure that CloudFormation succeeds without errors. This takes about 15 minutes.

Step 5: Follow the progress of the Qumulo provisioning node

1. Sign in to the AWS Management Console and open the Amazon EC2 console.

2. Select Instances in the left navigation pane.

3. Find the instance with Qumulo Provisioning Node in its name.

   

4. Monitor the Qumulo provisioning node until it transitions to the Stopped state.

It will start up, initialize, and run updates on all cluster nodes. Once it has finished running the updates, it will automatically stop.

Step 6: Post deployment tasks

There are several things you need to do after you’ve deployed Qumulo to ensure that it works with your studio.

Find the QumuloNode IPv4 address

First, you’ll need to find the IPv4 address of one of the QumuloNode instances. This address will be used to configure the DNS and Qumulo.

1. Sign in to the AWS Management Console and open the Amazon EC2 console.

2. Select Instances in the left navigation pane.

3. Select one of the deployed Qumulo Nodes; its name will start with QumuloNode and have an index as a suffix. It doesn’t matter which Qumulo Node you select.

4. In the details, copy the Private IPv4 address. This address will be used later in a PowerShell command to access the Qumulo dashboard.

   

Configure the DNS

Before joining Qumulo to your AWS Managed Microsoft AD, you need to change the DNS settings for Qumulo.

1. Launch a Windows-Default Nimble Workstation as an admin by following the Launching a virtual workstation.

2. Install the latest version of Python.

3. Select the Start Menu and search for cmd.

4. Open the context menu (right-click) on Command Prompt, then choose Run as administrator.

   

5. Run the following commands one line at a time.

SET IP=<qumulo cluster node private IPv4 address>

Note

The <qumulo cluster nodes private IPv4 address> is the Private IPv4 address that you found in Find the QumuloNode IPv4 address.

cd c:\Users\Admin\Desktop

mkdir qumulo & cd qumulo

python -m venv env

.\env\scripts\activate

pip install qumulo_api

qq --host %IP% version

qq --host %IP% login -u admin

qq --host %IP% network_mod_network --network-id 1 --dns-servers <first ip>,<second ip>

Note

<first ip> and <second ip> are the two Directory DNS addresses that you found in Find the Active Directory information.

qq --host %IP% network_get_network --network-id 1

Configure Qumulo

Before configuring the AWS Managed Microsoft AD, you need to log in to the Qumulo console and change the admin password.

1. Open a web browser in the workstation.

2. Go to https://qumulo_cluster_node_private_IPv4_address;

   1. The qumulo_cluster_node_private_IPv4_address is the Private IPv4 address that you found in Find the QumuloNode IPv4 address.

3. The web browser will display a warning stating: “The web browser doesn't have a cert for this website”. This warning is displayed because your workstation uses an internal IP address, and the Qumulo web console uses TLS (HTTPS) for security.

   1. Select Advanced.

   2. Select Accept the Risk.

   3. Select Continue.

      Important

      In general, don’t ignore website warnings. You should understand the possible consequences of visiting the website before accepting the risk.

4. Log in with the username Admin name and the password you chose when you deployed with CloudFormation in Step 4: Deploy Qumulo using CloudFormation.

You're now logged into the Qumulo Web Interface. This is where you can perform actions, such as configuring the AWS Managed Microsoft AD and creating file shares.

Configure the Active Directory

Next, you’ll use the admin account to join the Qumulo cluster to the AWS Managed Microsoft AD domain.

1. Still within the Qumulo web interface, select Cluster → Active Directory.

   

2. Enter the following Parameters.

   1. Domain Name: Use the Directory DNS name that you found in Find the Active Directory information.

   2. Domain User Name: Admin

   3. Domain Password: Use your Nimble Studio admin password

   4. NetBIOS Doman Name: Enter the Directory NetBIOS name that you found in Find the Active Directory information.

   5. Use Active Directory as your primary time server: Yes

   6. Use Active Directory for POSIX attributes: Yes

   7. Base DN: OU=Users,OU=<NetBIOS>,DC=<NetBIOS>,DC=nimble,DC=us-west-2,DC=aws

      1. For <NetBIOS>, use the Directory NetBIOS name that you found in Find the Active Directory information.

   8. Select JOIN.

Create file shares

Next, you’ll create an NFS export and an SMB share for Qumulo.

1. Go to Sharing → NFS Exports.

   

2. Delete the existing default entry.

   

3. Select Create.

4. In the Export path section, enter /Files.

5. Select Save.

6. Go to Sharing → SMB Shares.

7. Delete the existing default entry.

   

8. Select Create Share.

9. In the Share name section, enter Files.

10. In the Share Permissions section, check the box next to Write and the box next to Change permissions for the Everyone Trustee.

    

11. Select Create Share.

Configure a round-robin DNS alias for Qumulo

To distribute the client connection load evenly to the backend file server nodes, you need to set up a round robin DNS entry for the Qumulo nodes, and use that in your mount commands.

1. Select the Start Menu and search for PowerShell.

2. Open the context menu (right-click) on Windows PowerShell, then choose Run as administrator.

3. Run the following command: Install-WindowsFeature -Name DNS -IncludeManagementTools

   

4. Select the Start Menu and search for DNS.

5. Right-click on it and choose Run as administrator.

   

6. Select View → Advanced.

   

7. Select Action → Connect to DNS Server.

   

8. Select The Following Computer.

9. Enter one of the Directory DNS addresses of your Active Directory that you found in Find the Active Directory information.

10. Choose OK.

11. Select <Directory DNS address> → Forward Lookup Zones.

12. Select the entry which has the full domain name. Example: <your-studio-name>.nimble.<region>.aws

13. On the right, you will see four entries with the name cloud-q. Delete all of them.

    

14. On your local machine, go to Services → CloudFormation.

15. Choose the stack containing -QSTACK-.

16. Select the Outputs tab.

17. Notice the values from ClusterSecondaryPrivateIPs. This will be used to create new entries for all floating IPs for the Qumulo cluster.

    

18. Back in the workstation, select the Start Menu and search for PowerShell.

19. Open the context menu (right-click) on Windows PowerShell, then choose Run as administrator.

20. Run the following code sample to create new entries for all floating IPs for the Qumulo cluster.

    1. Replace $entries with the values of ClusterSecondaryPrivateIPs.

    2. Replace $dns_ip with one of the Directory DNS addresses of your Active Directory that you found in Find the Active Directory information.

    3. Replace -ZoneName with the Directory DNS name that you found in Find the Active Directory information.

       $entries = "<ClusterSecondaryPrivateIP>", "<ClusterSecondaryPrivateIP>", "<ClusterSecondaryPrivateIP>", "<ClusterSecondaryPrivateIP>", "<ClusterSecondaryPrivateIP>", "<ClusterSecondaryPrivateIP>", "<ClusterSecondaryPrivateIP>", "<ClusterSecondaryPrivateIP>", "<ClusterSecondaryPrivateIP>", "<ClusterSecondaryPrivateIP>", "<ClusterSecondaryPrivateIP>", "<ClusterSecondaryPrivateIP>"
           $dns_ip = "<dns-ip>"
           Foreach ($entry in $entries)
           {
               echo $entry
               Add-DnsServerResourceRecordA -Name cloud-q -ComputerName $dns_ip -IPv4Address $entry -ZoneName <Directory-DNS-name> -TimeToLive 00:00:00 -AgeRecord -AllowUpdateAny
           }

21. Run the following code: ipconfig /flushdns

22. Disconnect from, and terminate your workstation.

Create a Qumulo File Storage Component in Nimble Studio

For your streaming workstations and workers to access Qumulo, you need to add a new Nimble Studio component. The following instructions will show you how.

1. Go to Services → Nimble Studio .

2. Select Studio resources in the left navigation pane.

3. Under File storage, select Add.

4. Provide the following information in the specified fields.

   1. File storage name: (recommended) Qumulo

   2. Storage type: Custom

   3. Linux mount point: (recommended) /mnt/qumulo

   4. Windows mount drive: (recommended) Q

   5. Share: Files

      Note

      The value of Share must be the same as the Share name of the SMB share that you created in Create file shares.

   6. Endpoint: cloud-q.<domain name>

      1. Replace <doman name> with the Directory DNS name that you found in Find the Active Directory information. Example: cloud-q.<your studio name>.nimble.<region>.aws

         

5. Enter the following scripts in their corresponding sections.

   1. Windows system initialization script

      if ($authMode -eq "USERNAME_PASSWORD") {
          net use $windowsMountDrive':' \\$endpoint\$shareName /user:$username $password /persistent:no
      }

   2. Windows user initialization script

      if ($authMode -eq "KERBEROS") {
          net use $windowsMountDrive':' \\$endpoint\$shareName /persistent:no
      }

   3. Linux system initialization script

      mkdir -p $linuxMountPoint
      mountpoint -q $linuxMountPoint || mount -t nfs -o intr,hard,tcp $endpoint:/$shareName $linuxMountPoint

      

6. In the Security groups table, find and select the WorkstationEgress entry.

7. Read the terms and conditions at the bottom of the page and if you agree:

   1. Select the check box next to I understand that Nimble Studio will access my existing file storage.

8. Choose Save connection parameters.

   

Add the new component to launch profiles

You can attach the Qumulo component to any launch profile that you choose. At the minimum, we recommend that you attach it to the Workstation-Default and the RenderWorker-Default launch profiles.

1. Select Launch Profiles.

2. Select the Workstation-Default launch profile.

3. Select Action → Edit.

4. Make sure that Qumulo is selected under Studio file storage components in the Launch profile components section.

   

5. Read the terms and conditions at the bottom of the page and if you agree:

   1. Select the check box next to I understand that Nimble Studio will access my existing file storage.

6. Choose Save connection parameters.

   

7. Repeat steps 2-6 for the RenderWorker-Default launch profile.

   Note

   Only new streaming instances and workers will use this new component. Terminate and recreate streaming sessions and render worker instances as needed.

With this new custom file system studio component attached to your launch profile, streaming instances and workers that use this launch profile should automatically mount Qumulo on their next boot.