Amazon FSx for Windows File Server
Windows User Guide

Scaling Out Performance with Shards

Amazon FSx for Windows File Server supports the use of the Microsoft Distributed File System (DFS). By using DFS Namespaces, you can scale out performance (both read and write) to serve I/O-intensive workloads by spreading your file data across multiple Amazon FSx file systems. At the same time, you can still present a unified view under a common namespace to your applications. This solution involves dividing your file data into smaller datasets or shards and storing them across different file systems. Applications accessing your data from multiple instances can achieve high levels of performance by reading and writing to these shards in parallel.

You can use this solution when your workload requires uniformly distributed read/write access to your file data (for example, if each subset of compute instances accesses a different portion of your file data). Otherwise, if you need a solution targeted at scale-out read-heavy workloads that need access to the same file dataset (for example, workloads for data analytics, video transcoding, machine learning, and artificial intelligence), see Scaling Out Read Performance with Read Replicas.

Setting Up DFS Namespaces for Scale-Out Performance

The following procedure guides you through creating a DFS solution on Amazon FSx for scale-out performance. In this example, the data stored in the corp namespace is sharded alphabetically. Data files ‘A-F’, ‘G-M’ and ‘N-Z’ are all stored on different file shares. Based on the type of data, I/O size, and I/O access pattern, you should decide how to best shard your data across multiple file shares. Choose a sharding convention that distributes I/O evenly across all the file shares you plan on using. Keep in mind that each namespace supports up to 50,000 file shares and hundreds of petabytes of storage capacity in aggregate.

To set up DFS Namespaces for scale-out performance

  1. If you don't already have DFS Namespace servers running, you can launch a pair of highly available DFS Namespace servers using the setup-DFSN-servers.template AWS CloudFormation template. For more information on creating an AWS CloudFormation stack, see Creating a Stack on the AWS CloudFormation Console in the AWS CloudFormation User Guide.

  2. Connect to one of the DFS Namespace servers launched in the previous step as a user in the AWS Delegated Administrators group. For more information, see Connecting to Your Windows Instance in the Amazon EC2 User Guide for Windows Instances.

  3. Access the DFS Management Console. Open the Start menu and run dfsmgmt.msc. This opens the DFS Management GUI tool.

  4. Choose Action then New Namespace, type in the computer name of the first DFS Namespace server you launched for Server and choose Next.

  5. For Name, type in the namespace you're creating (for example, corp).

  6. Choose Edit Settings and set the appropriate permissions based on your requirements. Choose Next.

  7. Leave the default Domain-based namespace option selected, leave the Enable Windows Server 2008 mode option selected, and choose Next.

    Note

    Windows Server 2008 mode is the latest available option for Namespaces.

  8. Review the namespace settings and choose Create.

  9. With the newly created namespace selected under Namespaces in the navigation bar, choose Action then Add Namespace Server.

  10. Type in the computer name of the second DFS Namespace server you launched for Namespace server.

  11. Choose Edit Settings, set the appropriate permissions based on your requirements, and choose OK.

  12. Open the context (right-click) menu for the namespace you just created, choose New Folder, enter the name of the folder for the first shard (for example, A-F for Name), and choose Add.

  13. Type in the DNS name of the file share hosting this shard in UNC format (for example, \\fs-0123456789abcdef0.example.com\A-F) for Path to folder target and choose OK.

  14. If the share doesn't exist:

    1. Choose Yes to create it.

    2. From the Create Share dialog, choose Browse.

    3. Choose an existing folder, or create a new folder under D$, and choose OK.

    4. Set the appropriate share permissions, and choose OK.

  15. With the folder target now added for the shard, choose OK.

  16. Repeat the last four steps for other shards you want to add to the same namespace.