Automatically export updates to your S3 bucket - FSx for Lustre

Automatically export updates to your S3 bucket

You can configure your FSx for Lustre file system to automatically update the contents of a linked S3 bucket as files are added, changed, or deleted on the file system. FSx for Lustre creates, updates, or deletes the object in S3, corresponding to the change in the file system.

Note

Automatic export isn't available on FSx for Lustre 2.10 file systems or Scratch 1 file systems.

You can export to a data repository that is in the same AWS Region as the file system or in a different AWS Region.

You can configure automatic export when you create the data repository association and update the automatic export settings at any time using the FSx management console, the AWS CLI, and the AWS API.

Note

You can configure both automatic export and automatic import on the same data repository association. This topic describes only the automatic export feature.

Important
  • If a file is modified in the file system with all automatic export policies enabled and automatic import disabled, the content of that file is always exported to a corresponding object in S3. If an object already exists in the target location, the object is overwritten.

  • If a file is modified in both the file system and S3, with all automatic import and automatic export policies enabled, either the file in the file system or the object in S3 could be overwritten by the other. It isn't guaranteed that a later edit in one location will overwrite an earlier edit in another location. If you modify the same file in both the file system and the S3 bucket, you should ensure application-level coordination to prevent such conflicts. FSx for Lustre doesn't prevent conflicting writes in multiple locations.

The export policy specifies how you want FSx for Lustre to update your linked S3 bucket as the contents change in the file system. A data repository association can have one of the following automatic export policies:

  • New – FSx for Lustre automatically updates the S3 data repository only when a new file, directory, or symlink is created on the file system.

  • Changed – FSx for Lustre automatically updates the S3 data repository only when an existing file in the file system is changed. For file content changes, the file must be closed before it's propagated to the S3 repository. Metadata changes (rename, ownership, permissions, and timestamps) are propagated when the operation is done. For renaming changes (including moves), the existing (pre-renamed) S3 object is deleted and a new S3 object is created with the new name.

  • Deleted – FSx for Lustre automatically updates the S3 data repository only when a file, directory, or symlink is deleted in the file system.

  • Any combination of New, Changed, and Deleted – FSx for Lustre automatically updates the S3 data repository when any of the specified actions occur in file system. For example, you can specify that the S3 repository is updated when a file is added to (New) or removed from (Deleted) the file system, but not when a file is changed.

  • No policy configured – FSx for Lustre doesn't automatically update the S3 data repository when files are added to, changed in, or deleted from the file system. If you don't configure an export policy, automatic export is disabled. You can still manually export changes by using an export data repository task, as described in Using data repository tasks to export changes.

For most use cases, we recommend that you configure an export policy of New, Changed, and Deleted. This policy ensures that all updates made on your file system are automatically exported to your linked S3 data repository.

We recommend that you turn on logging to CloudWatch Logs to log information about any files or directories that couldn't be exported automatically. Warnings and errors in the log contain information about the failure reason. For more information, see Data repository event logs.

Updating export settings

You can set a file system's export settings to a linked S3 bucket when you create the data repository association. For more information, see Creating a link to an S3 bucket.

You can also update the export settings at any time, including the export policy. For more information, see Updating data repository association settings.

Monitoring automatic export

You can monitor automatic export enabled data repository associations using a set of metrics published to Amazon CloudWatch. The AgeOfOldestQueuedMessage metric represents the age of the oldest update made to the file system which has not yet been exported to S3. If the AgeOfOldestQueuedMessage is greater than zero for an extended period of time, we recommend temporarily reducing the number of changes (directory renames in particular) that are actively being made to the file system until the message queue has been reduced. For more information, see AutoImport and AutoExport metrics.

Important

When deleting a data repository association or file system with automatic export enabled, you should first make sure that AgeOfOldestQueuedMessage is zero, meaning that there are no changes that have not yet been exported. If AgeOfOldestQueuedMessage is greater than zero when you delete your data repository association or file system, the changes that had not yet been exported will not reach your linked S3 bucket. To avoid this, wait for AgeOfOldestQueuedMessage to reach zero before deleting your data repository association or file system.