Working with older deployment types
This section applies to file systems with Scratch 1 deployment type, and also to file systems
with Scratch 2 or Persistent 1 deployment types that do not use
data repository associations.
Link your file system to an Amazon S3 bucket
When you create an Amazon FSx for Lustre file system, you can link it to a durable data repository in Amazon S3. Before you create your file system, make sure that you have already created the Amazon S3 bucket to which you are linking. In the Create file system wizard, you set the following data repository configuration properties in the optional Data Repository Import/Export pane.
-
Choose how Amazon FSx keeps your file and directory listing up to date as you add or modify objects in your S3 bucket after the file system is created. For more information, see Automatically import updates from your S3 bucket.
-
Import bucket: Enter the name of the S3 bucket that you are using for the linked repository.
-
Import prefix: Enter an optional import prefix if you want to import only some file and directory listings of data in your S3 bucket into your file system. The import prefix defines where in your S3 bucket to import data from.
-
Export prefix: Defines where Amazon FSx exports the contents of your file system to your linked S3 bucket.
You can have a 1:1 mapping where Amazon FSx exports data from your FSx for Lustre file system back to the same directories on the S3 bucket that it was imported from. To have a 1:1 mapping, specify an export path to the S3 bucket without any prefixes when you create your file system.
-
When you create file system using the console, choose Export prefix > A prefix you specify option, and keep the prefix field blank.
-
When you create a file system using the AWS CLI or API, specify the export path as the name of the S3 bucket without any additional prefixes, for example,
ExportPath=s3://lustre-export-test-bucket/.
Using this method, you can include an import prefix when you specify the import path, and it doesn't impact a 1:1 mapping for exports.
Creating file systems linked to an S3 bucket
The following procedures walk you through the process of creating an Amazon FSx file system linked to an S3 bucket using the AWS Management Console and AWS Command Line Interface (AWS CLI).
Viewing a file system's export path
You can view a file system's export path using the FSx for Lustre console, the AWS CLI, and the API.
Data repository lifecycle state
The data repository lifecycle state provides status information about the file system's linked data repository. A data repository can have the following Lifecycle states.
-
Creating: Amazon FSx is creating the data repository configuration between the file system and the linked data repository. The data repository is unavailable.
-
Available: The data repository is available for use.
-
Updating: The data repository configuration is undergoing a customer-initiated update that might affect its availability.
-
Misconfigured: Amazon FSx cannot automatically import updates from the S3 bucket until the data repository configuration is corrected. For more information, see Troubleshooting a misconfigured linked S3 bucket.
You can view a file system's linked data repository lifecycle state using the Amazon FSx console, the AWS Command Line Interface, and the Amazon FSx API.
In the Amazon FSx console, you can access the data repository Lifecycle state in the Data Repository Integration
pane of the Data Repository tab for the file system.
The Lifecycle property is located in the DataRepositoryConfiguration object in the response of a
describe-file-systems CLI command
(the equivalent API action is DescribeFileSystems).
Automatically import updates from your S3 bucket
By default, when you create a new file system, Amazon FSx imports the file metadata (the name, ownership, timestamp, and permissions) of objects in the linked S3 bucket at file system creation. You can configure your FSx for Lustre file system to automatically import metadata of objects that are added to, changed in, or deleted from your S3 bucket after file system creation. FSx for Lustre updates the file and directory listing of a changed object after creation in the same manner as it imports file metadata at file system creation. When Amazon FSx updates the file and directory listing of a changed object, if the changed object in the S3 bucket no longer contains its metadata, Amazon FSx maintains the current metadata values of the file, rather than using default permissions.
Note
Import settings are available on FSx for Lustre file systems created after 3:00 pm EDT, July 23, 2020.
You can set import preferences when you create a new file system, and you can update the setting on existing file systems using the FSx management console, the AWS CLI, and the AWS API. When you create your file system, your existing S3 objects appear as file and directory listings. After you create your file system, how do you want to update it as the contents of your S3 bucket are updated? A file system can have one of the following Import preferences:
Note
The FSx for Lustre file system and its linked S3 bucket must be located in the same AWS Region to automatically import updates.
-
Update my file and directory listing as objects are added to my S3 bucket: (Default) Amazon FSx automatically updates file and directory listings of any new objects added to the linked S3 bucket that do not currently exist in the FSx file system. Amazon FSx does not update listings for objects that have changed in the S3 bucket. Amazon FSx does not delete listings of objects that are deleted in the S3 bucket.
Note
The default import preferences setting for importing data from a linked S3 bucket using the CLI and API is
NONE. The default import preferences setting when using the console is to update Lustre as new objects are added to the S3 bucket. -
Update my file and directory listing as objects are added to or changed in my S3 bucket: Amazon FSx automatically updates file and directory listings of any new objects added to the S3 bucket and any existing objects that are changed in the S3 bucket after you choose this option. Amazon FSx does not delete listings of objects that are deleted in the S3 bucket.
-
Update my file and directory listing as objects are added to, changed in, or deleted from my S3 bucket: Amazon FSx automatically updates file and directory listings of any new objects added to the S3 bucket, any existing objects that are changed in the S3 bucket, and any existing objects that are deleted in the S3 bucket after you choose this option.
-
Do not update my file and directly listing when objects are added to, changed in, or deleted from my S3 bucket - Amazon FSx only updates file and directory listings from the linked S3 bucket when the file system is created. FSx does not update file and directory listings for any new, changed, or deleted objects after choosing this option.
When you set the import preferences to update your file system file and directory listings
based on changes in the linked S3 bucket, Amazon FSx creates an event notification
configuration on the linked S3 bucket named FSx. Do not modify or delete the FSx event
notification configuration on the S3 bucket—doing so prevents the automatic import of
new or changed file and directory listings to your file system.
When Amazon FSx updates a file listing that has changed on the linked S3 bucket, it overwrites the local file with the updated version, even if the file is write-locked. Similarly, when Amazon FSx updates a file listing when the corresponding object has been deleted on the linked S3 bucket, it deletes the local file, even if the file is write- locked.
Amazon FSx makes a best effort to update your file system. Amazon FSx cannot update the file system with changes in the following situations:
-
When Amazon FSx does not have permission to open the changed or new S3 object.
-
When the
FSxevent notification configuration on the linked S3 bucket is deleted or changed.
Either of these conditions cause the data repository lifecycle state to become Misconfigured. For more information, see Data repository lifecycle state.
Prerequisites
The following conditions are required for Amazon FSx to automatically import new, changed, or deleted files from the linked S3 bucket:
-
The file system and its linked S3 bucket must be located in the same AWS Region.
-
The S3 bucket does not have a misconfigured Lifecycle state. For more information, see Data repository lifecycle state.
-
Your account must have the permissions required to configure and receive event notifications on the linked S3 bucket.
Types of file changes supported
Amazon FSx supports importing the following changes to files and folders that occur in the linked S3 bucket:
-
Changes to file contents
-
Changes to file or folder metadata
-
Changes to symlink target or metadata
Updating import preferences
You can set a file system's import preferences when you create a new file system. For more information, see Linking your file system to an S3 bucket.
You can also update a file system's import preferences after it is created using the AWS Management Console, the AWS CLI, and the Amazon FSx API, as shown in the following procedure.
