Namespace Amazon.CDK.AWS.FSx
Amazon FSx Construct Library
Amazon FSx provides fully managed third-party file systems with the native compatibility and feature sets for workloads such as Microsoft Windows–based storage, high-performance computing, machine learning, and electronic design automation.
Amazon FSx supports two file system types: Lustre and Windows File Server.
FSx for Lustre
Amazon FSx for Lustre makes it easy and cost-effective to launch and run the popular, high-performance Lustre file system. You use Lustre for workloads where speed matters, such as machine learning, high performance computing (HPC), video processing, and financial modeling.
The open-source Lustre file system is designed for applications that require fast storage—where you want your storage to keep up with your compute. Lustre was built to solve the problem of quickly and cheaply processing the world's ever-growing datasets. It's a widely used file system designed for the fastest computers in the world. It provides submillisecond latencies, up to hundreds of GBps of throughput, and up to millions of IOPS. For more information on Lustre, see the Lustre website.
As a fully managed service, Amazon FSx makes it easier for you to use Lustre for workloads where storage speed matters. Amazon FSx for Lustre eliminates the traditional complexity of setting up and managing Lustre file systems, enabling you to spin up and run a battle-tested high-performance file system in minutes. It also provides multiple deployment options so you can optimize cost for your needs.
Amazon FSx for Lustre is POSIX-compliant, so you can use your current Linux-based applications without having to make any changes. Amazon FSx for Lustre provides a native file system interface and works as any file system does with your Linux operating system. It also provides read-after-write consistency and supports file locking.
Installation
Import to your project:
using Amazon.CDK.AWS.FSx;
Basic Usage
Setup required properties and create:
Vpc vpc;
var fileSystem = new LustreFileSystem(this, "FsxLustreFileSystem", new LustreFileSystemProps {
LustreConfiguration = new LustreConfiguration { DeploymentType = LustreDeploymentType.SCRATCH_2 },
StorageCapacityGiB = 1200,
Vpc = vpc,
VpcSubnet = vpc.PrivateSubnets[0]
});
Connecting
To control who can access the file system, use the .connections
attribute. FSx has a fixed default port, so you don't
need to specify the port. This example allows an EC2 instance to connect to a file system:
LustreFileSystem fileSystem;
Instance instance;
fileSystem.Connections.AllowDefaultPortFrom(instance);
Mounting
The LustreFileSystem Construct exposes both the DNS name of the file system as well as its mount name, which can be used to mount the file system on an EC2 instance. The following example shows how to bring up a file system and EC2 instance, and then use User Data to mount the file system on the instance at start-up:
using Amazon.CDK.AWS.IAM;
Vpc vpc;
IDictionary<string, LustreDeploymentType> lustreConfiguration = new Dictionary<string, LustreDeploymentType> {
{ "deploymentType", LustreDeploymentType.SCRATCH_2 }
};
var fs = new LustreFileSystem(this, "FsxLustreFileSystem", new LustreFileSystemProps {
LustreConfiguration = lustreConfiguration,
StorageCapacityGiB = 1200,
Vpc = vpc,
VpcSubnet = vpc.PrivateSubnets[0]
});
var inst = new Instance(this, "inst", new InstanceProps {
InstanceType = InstanceType.Of(InstanceClass.T2, InstanceSize.LARGE),
MachineImage = new AmazonLinuxImage(new AmazonLinuxImageProps {
Generation = AmazonLinuxGeneration.AMAZON_LINUX_2
}),
Vpc = vpc,
VpcSubnets = new SubnetSelection {
SubnetType = SubnetType.PUBLIC
}
});
fs.Connections.AllowDefaultPortFrom(inst);
// Need to give the instance access to read information about FSx to determine the file system's mount name.
inst.Role.AddManagedPolicy(ManagedPolicy.FromAwsManagedPolicyName("AmazonFSxReadOnlyAccess"));
var mountPath = "/mnt/fsx";
var dnsName = fs.DnsName;
var mountName = fs.MountName;
inst.UserData.AddCommands("set -eux", "yum update -y", "amazon-linux-extras install -y lustre2.10", $"mkdir -p {mountPath}", $"chmod 777 {mountPath}", $"chown ec2-user:ec2-user {mountPath}", $"echo \"{dnsName}@tcp:/{mountName} {mountPath} lustre defaults,noatime,flock,_netdev 0 0\" >> /etc/fstab", "mount -a");
Importing an existing Lustre filesystem
An FSx for Lustre file system can be imported with fromLustreFileSystemAttributes(this, id, attributes)
. The
following example lays out how you could import the SecurityGroup a file system belongs to, use that to import the file
system, and then also import the VPC the file system is in and add an EC2 instance to it, giving it access to the file
system.
var sg = SecurityGroup.FromSecurityGroupId(this, "FsxSecurityGroup", "{SECURITY-GROUP-ID}");
var fs = LustreFileSystem.FromLustreFileSystemAttributes(this, "FsxLustreFileSystem", new FileSystemAttributes {
DnsName = "{FILE-SYSTEM-DNS-NAME}",
FileSystemId = "{FILE-SYSTEM-ID}",
SecurityGroup = sg
});
var vpc = Vpc.FromVpcAttributes(this, "Vpc", new VpcAttributes {
AvailabilityZones = new [] { "us-west-2a", "us-west-2b" },
PublicSubnetIds = new [] { "{US-WEST-2A-SUBNET-ID}", "{US-WEST-2B-SUBNET-ID}" },
VpcId = "{VPC-ID}"
});
var inst = new Instance(this, "inst", new InstanceProps {
InstanceType = InstanceType.Of(InstanceClass.T2, InstanceSize.LARGE),
MachineImage = new AmazonLinuxImage(new AmazonLinuxImageProps {
Generation = AmazonLinuxGeneration.AMAZON_LINUX_2
}),
Vpc = vpc,
VpcSubnets = new SubnetSelection {
SubnetType = SubnetType.PUBLIC
}
});
fs.Connections.AllowDefaultPortFrom(inst);
Lustre Data Repository Association support
The LustreFilesystem Construct supports one Data Repository Association (DRA) to an S3 bucket. This allows Lustre hierarchical storage management to S3 buckets, which in turn makes it possible to use S3 as a permanent backing store, and use FSx for Lustre as a temporary high performance cache.
Note: CloudFormation does not currently support for PERSISTENT_2
filesystems, and so neither does CDK.
The following example illustrates setting up a DRA to an S3 bucket, including automated metadata import whenever a file is changed, created or deleted in the S3 bucket:
using Amazon.CDK.AWS.S3;
Vpc vpc;
Bucket bucket;
IDictionary<string, object> lustreConfiguration = new Dictionary<string, object> {
{ "deploymentType", LustreDeploymentType.SCRATCH_2 },
{ "exportPath", bucket.S3UrlForObject() },
{ "importPath", bucket.S3UrlForObject() },
{ "autoImportPolicy", LustreAutoImportPolicy.NEW_CHANGED_DELETED }
};
var fs = new LustreFileSystem(this, "FsxLustreFileSystem", new LustreFileSystemProps {
Vpc = vpc,
VpcSubnet = vpc.PrivateSubnets[0],
StorageCapacityGiB = 1200,
LustreConfiguration = lustreConfiguration
});
Compression
By default, transparent compression of data within FSx for Lustre is switched off. To enable it, add the following to your lustreConfiguration
:
IDictionary<string, LustreDataCompressionType> lustreConfiguration = new Dictionary<string, LustreDataCompressionType> {
// ...
{ "dataCompressionType", LustreDataCompressionType.LZ4 }
};
When you turn data compression on for an existing file system, only newly written files are compressed. Existing files are not compressed. For more information, see Compressing previously written files.```
FSx for Windows File Server
The L2 construct for the FSx for Windows File Server has not yet been implemented. To instantiate an FSx for Windows file system, the L1 constructs can be used as defined by CloudFormation.
Classes
CfnDataRepositoryAssociation | Creates an Amazon FSx for Lustre data repository association (DRA). |
CfnDataRepositoryAssociation.AutoExportPolicyProperty | Describes a data repository association's automatic export policy. |
CfnDataRepositoryAssociation.AutoImportPolicyProperty | Describes the data repository association's automatic import policy. |
CfnDataRepositoryAssociation.S3Property | The configuration for an Amazon S3 data repository linked to an Amazon FSx Lustre file system with a data repository association. |
CfnDataRepositoryAssociationProps | Properties for defining a |
CfnFileSystem | The |
CfnFileSystem.AuditLogConfigurationProperty | The configuration that Amazon FSx for Windows File Server uses to audit and log user accesses of files, folders, and file shares on the Amazon FSx for Windows File Server file system. |
CfnFileSystem.ClientConfigurationsProperty | Specifies who can mount an OpenZFS file system and the options available while mounting the file system. |
CfnFileSystem.DiskIopsConfigurationProperty | The SSD IOPS (input/output operations per second) configuration for an Amazon FSx for NetApp ONTAP, Amazon FSx for Windows File Server, or FSx for OpenZFS file system. |
CfnFileSystem.LustreConfigurationProperty | The configuration for the Amazon FSx for Lustre file system. |
CfnFileSystem.NfsExportsProperty | The configuration object for mounting a file system. |
CfnFileSystem.OntapConfigurationProperty | The configuration for this Amazon FSx for NetApp ONTAP file system. |
CfnFileSystem.OpenZFSConfigurationProperty | The OpenZFS configuration for the file system that's being created. |
CfnFileSystem.RootVolumeConfigurationProperty | The configuration of an Amazon FSx for OpenZFS root volume. |
CfnFileSystem.SelfManagedActiveDirectoryConfigurationProperty | The configuration that Amazon FSx uses to join a FSx for Windows File Server file system or an FSx for ONTAP storage virtual machine (SVM) to a self-managed (including on-premises) Microsoft Active Directory (AD) directory. |
CfnFileSystem.UserAndGroupQuotasProperty | Used to configure quotas that define how much storage a user or group can use on an FSx for OpenZFS volume. |
CfnFileSystem.WindowsConfigurationProperty | The Microsoft Windows configuration for the file system that's being created. |
CfnFileSystemProps | Properties for defining a |
CfnSnapshot | A snapshot of an Amazon FSx for OpenZFS volume. |
CfnSnapshotProps | Properties for defining a |
CfnStorageVirtualMachine | Creates a storage virtual machine (SVM) for an Amazon FSx for ONTAP file system. |
CfnStorageVirtualMachine.ActiveDirectoryConfigurationProperty | Describes the self-managed Microsoft Active Directory to which you want to join the SVM. |
CfnStorageVirtualMachine.SelfManagedActiveDirectoryConfigurationProperty | The configuration that Amazon FSx uses to join the ONTAP storage virtual machine (SVM) to your self-managed (including on-premises) Microsoft Active Directory directory. |
CfnStorageVirtualMachineProps | Properties for defining a |
CfnVolume | Creates an FSx for ONTAP or Amazon FSx for OpenZFS storage volume. |
CfnVolume.AggregateConfigurationProperty | Use to specify configuration options for a volume’s storage aggregate or aggregates. |
CfnVolume.AutocommitPeriodProperty | Sets the autocommit period of files in an FSx for ONTAP SnapLock volume, which determines how long the files must remain unmodified before they're automatically transitioned to the write once, read many (WORM) state. |
CfnVolume.ClientConfigurationsProperty | Specifies who can mount an OpenZFS file system and the options available while mounting the file system. |
CfnVolume.NfsExportsProperty | The configuration object for mounting a Network File System (NFS) file system. |
CfnVolume.OntapConfigurationProperty | Specifies the configuration of the ONTAP volume that you are creating. |
CfnVolume.OpenZFSConfigurationProperty | Specifies the configuration of the Amazon FSx for OpenZFS volume that you are creating. |
CfnVolume.OriginSnapshotProperty | The configuration object that specifies the snapshot to use as the origin of the data for the volume. |
CfnVolume.RetentionPeriodProperty | Specifies the retention period of an FSx for ONTAP SnapLock volume. |
CfnVolume.SnaplockConfigurationProperty | Specifies the SnapLock configuration for an FSx for ONTAP SnapLock volume. |
CfnVolume.SnaplockRetentionPeriodProperty | The configuration to set the retention period of an FSx for ONTAP SnapLock volume. |
CfnVolume.TieringPolicyProperty | Describes the data tiering policy for an ONTAP volume. |
CfnVolume.UserAndGroupQuotasProperty | Configures how much storage users and groups can use on the volume. |
CfnVolumeProps | Properties for defining a |
FileSystemAttributes | Properties that describe an existing FSx file system. |
FileSystemBase | A new or imported FSx file system. |
FileSystemProps | Properties for the FSx file system. |
LustreAutoImportPolicy | The different auto import policies which are allowed. |
LustreConfiguration | The configuration for the Amazon FSx for Lustre file system. |
LustreDataCompressionType | The permitted Lustre data compression algorithms. |
LustreDeploymentType | The different kinds of file system deployments used by Lustre. |
LustreFileSystem | The FSx for Lustre File System implementation of IFileSystem. |
LustreFileSystemProps | Properties specific to the Lustre version of the FSx file system. |
LustreMaintenanceTime | Class for scheduling a weekly manitenance time. |
LustreMaintenanceTimeProps | Properties required for setting up a weekly maintenance time. |
Weekday | Enum for representing all the days of the week. |
Interfaces
CfnDataRepositoryAssociation.IAutoExportPolicyProperty | Describes a data repository association's automatic export policy. |
CfnDataRepositoryAssociation.IAutoImportPolicyProperty | Describes the data repository association's automatic import policy. |
CfnDataRepositoryAssociation.IS3Property | The configuration for an Amazon S3 data repository linked to an Amazon FSx Lustre file system with a data repository association. |
CfnFileSystem.IAuditLogConfigurationProperty | The configuration that Amazon FSx for Windows File Server uses to audit and log user accesses of files, folders, and file shares on the Amazon FSx for Windows File Server file system. |
CfnFileSystem.IClientConfigurationsProperty | Specifies who can mount an OpenZFS file system and the options available while mounting the file system. |
CfnFileSystem.IDiskIopsConfigurationProperty | The SSD IOPS (input/output operations per second) configuration for an Amazon FSx for NetApp ONTAP, Amazon FSx for Windows File Server, or FSx for OpenZFS file system. |
CfnFileSystem.ILustreConfigurationProperty | The configuration for the Amazon FSx for Lustre file system. |
CfnFileSystem.INfsExportsProperty | The configuration object for mounting a file system. |
CfnFileSystem.IOntapConfigurationProperty | The configuration for this Amazon FSx for NetApp ONTAP file system. |
CfnFileSystem.IOpenZFSConfigurationProperty | The OpenZFS configuration for the file system that's being created. |
CfnFileSystem.IRootVolumeConfigurationProperty | The configuration of an Amazon FSx for OpenZFS root volume. |
CfnFileSystem.ISelfManagedActiveDirectoryConfigurationProperty | The configuration that Amazon FSx uses to join a FSx for Windows File Server file system or an FSx for ONTAP storage virtual machine (SVM) to a self-managed (including on-premises) Microsoft Active Directory (AD) directory. |
CfnFileSystem.IUserAndGroupQuotasProperty | Used to configure quotas that define how much storage a user or group can use on an FSx for OpenZFS volume. |
CfnFileSystem.IWindowsConfigurationProperty | The Microsoft Windows configuration for the file system that's being created. |
CfnStorageVirtualMachine.IActiveDirectoryConfigurationProperty | Describes the self-managed Microsoft Active Directory to which you want to join the SVM. |
CfnStorageVirtualMachine.ISelfManagedActiveDirectoryConfigurationProperty | The configuration that Amazon FSx uses to join the ONTAP storage virtual machine (SVM) to your self-managed (including on-premises) Microsoft Active Directory directory. |
CfnVolume.IAggregateConfigurationProperty | Use to specify configuration options for a volume’s storage aggregate or aggregates. |
CfnVolume.IAutocommitPeriodProperty | Sets the autocommit period of files in an FSx for ONTAP SnapLock volume, which determines how long the files must remain unmodified before they're automatically transitioned to the write once, read many (WORM) state. |
CfnVolume.IClientConfigurationsProperty | Specifies who can mount an OpenZFS file system and the options available while mounting the file system. |
CfnVolume.INfsExportsProperty | The configuration object for mounting a Network File System (NFS) file system. |
CfnVolume.IOntapConfigurationProperty | Specifies the configuration of the ONTAP volume that you are creating. |
CfnVolume.IOpenZFSConfigurationProperty | Specifies the configuration of the Amazon FSx for OpenZFS volume that you are creating. |
CfnVolume.IOriginSnapshotProperty | The configuration object that specifies the snapshot to use as the origin of the data for the volume. |
CfnVolume.IRetentionPeriodProperty | Specifies the retention period of an FSx for ONTAP SnapLock volume. |
CfnVolume.ISnaplockConfigurationProperty | Specifies the SnapLock configuration for an FSx for ONTAP SnapLock volume. |
CfnVolume.ISnaplockRetentionPeriodProperty | The configuration to set the retention period of an FSx for ONTAP SnapLock volume. |
CfnVolume.ITieringPolicyProperty | Describes the data tiering policy for an ONTAP volume. |
CfnVolume.IUserAndGroupQuotasProperty | Configures how much storage users and groups can use on the volume. |
ICfnDataRepositoryAssociationProps | Properties for defining a |
ICfnFileSystemProps | Properties for defining a |
ICfnSnapshotProps | Properties for defining a |
ICfnStorageVirtualMachineProps | Properties for defining a |
ICfnVolumeProps | Properties for defining a |
IFileSystem | Interface to implement FSx File Systems. |
IFileSystemAttributes | Properties that describe an existing FSx file system. |
IFileSystemProps | Properties for the FSx file system. |
ILustreConfiguration | The configuration for the Amazon FSx for Lustre file system. |
ILustreFileSystemProps | Properties specific to the Lustre version of the FSx file system. |
ILustreMaintenanceTimeProps | Properties required for setting up a weekly maintenance time. |