# Monitoring Amazon FSx for Lustre file systems Monitoring is an important part of maintaining the reliability, availability, and performance of your FSx for Lustre file system and your other AWS solutions. Collecting monitoring data from all parts of your AWS solution allows you to more easily debug a multi-point failure if one occurs. You can monitor your FSx for Lustre file system, report when something is wrong, and take action automatically when appropriate using the following tools: + **Amazon CloudWatch** – Monitors your AWS resources and the applications that you run on AWS in real time. You can collect and track metrics, create customized dashboards, and set alarms that notify you when a specified metric reaches a threshold that you specify. For example, you can have CloudWatch track storage capacity or other metrics for your Amazon FSx for Lustre instances and automatically launch new instances when needed. + **Lustre logging** – Monitors the enabled logging events for your file system. Lustre logging writes these events to Amazon CloudWatch Logs. + **AWS CloudTrail** – Captures API calls and related events made by or on behalf of your AWS account and delivers the log files to an Amazon S3 bucket that you specify. You can identify which users and accounts called AWS, the source IP address from which the calls were made, and when the calls occurred. The following sections provide information on how to use the tools with your FSx for Lustre file systems. **Topics** + [Monitoring with Amazon CloudWatch](monitoring-cloudwatch.md) + [Logging with Amazon CloudWatch Logs](cw-event-logging.md) + [Logging FSx for Lustre API calls with AWS CloudTrail](logging-using-cloudtrail.md) # Monitoring with Amazon CloudWatch You can monitor Amazon FSx for Lustre using CloudWatch, which collects and processes raw data from Amazon FSx for Lustre into readable, near real-time metrics. These statistics are retained for a period of 15 months, so that you can access historical information and gain a better perspective on how your application or service is performing. For more information about CloudWatch, see [What is Amazon CloudWatch?](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) in the *Amazon CloudWatch User Guide*. CloudWatch metrics for FSx for Lustre are organized into six categories: + **Network I/O metrics** – Measure activity between clients and your file system. + **Object storage server metrics** – Measure object storage server (OSS) network throughput and disk throughput utilization. + **Object storage target metrics** – Measure object storage target (OST) disk throughput and disk IOPS utilization. + **Metadata metrics** – Measure metadata server (MDS) CPU utilization, metadata target (MDT) IOPS utilization, and client metadata operations. + **Storage capacity metrics** – Measure storage capacity utilization. + **S3 data repository metrics** – Measure age of oldest message waiting to be imported or exported, and renames processed by the file system. The following diagram illustrates an FSx for Lustre file system, its components, and its metric categories. ![\[FSx for Lustre reports metrics in CloudWatch.\]](http://docs.aws.amazon.com/fsx/latest/LustreGuide/images/metrics-overview.png) FSx for Lustre sends metric data to CloudWatch at 1-minute intervals. **Note** Metrics may not be published during file system maintenance windows for your Amazon FSx for Lustre file system. **Topics** + [How to use Amazon FSx for Lustre CloudWatch metrics](how_to_use_metrics.md) + [Accessing CloudWatch metrics](accessingmetrics.md) + [Amazon FSx for Lustre metrics and dimensions](fs-metrics.md) + [Performance warnings and recommendations](performance-insights.md) + [Creating CloudWatch alarms to monitor metrics](creating_alarms.md) # How to use Amazon FSx for Lustre CloudWatch metrics There are two primary architectural components of each Amazon FSx for Lustre file system: + One or more **object storage servers (OSSs)** that serve data to clients that access the file system. Each OSS is attached to one or more storage volumes, known as **object storage targets (OSTs)**, that host the data in your file system. + One or more **metadata servers (MDSs)** that serve metadata to clients that access the file system. Each MDS is attached to a storage volume, known as a **metadata target (MDT)**, that stores metadata such as filenames, directories, access permissions, and file layouts. FSx for Lustre reports metrics in CloudWatch that track performance and resource utilization for your file system's storage and metadata servers, and their associated storage volumes. The following diagram illustrates an Amazon FSx for Lustre file system with its architectural components, and the performance and resource CloudWatch metrics that are available for monitoring. ![\[Diagram displaying the different types of FSx for Lustre Cloudwatch metrics.\]](http://docs.aws.amazon.com/fsx/latest/LustreGuide/images/file-server-metrics.png) You can use the **Monitoring & performance** panel on your file system's dashboard in the Amazon FSx for Lustre console to view the metrics that are described in the following tables. For more information, see [Accessing CloudWatch metrics](accessingmetrics.md). **File system activity (in Summary tab)** | How do I... | Chart | Relevant metrics | | --- | --- | --- | | ...determine the amount of available storage capacity on my file system? | Available storage capacity (bytes) | `FreeDataStorageCapacity` | | ...determine my file system's total client throughput? | Total client throughput (bytes/sec) | SUM(`DataReadBytes` \$1 `DataWriteBytes`)/PERIOD (in seconds) | | …determine my file system’s total client IOPS? | Total client IOPS (operations/sec) | SUM(DataReadOperations \$1 DataWriteOperations \$1 MetadataOperations)/PERIOD (in seconds) | | ...determine the number of connections that are established between clients and my file server? | Client connections (count) | ClientConnections | | …determine my file system’s metadata performance utilization? | Metadata IOPS utilization (percent) | MAX(MDT Disk IOPS) | **Storage tab** | How do I... | Chart | Relevant metrics | | --- | --- | --- | | ...determine how much storage is available? | Available storage capacity (bytes) | `FreeDataStorageCapacity` | | …determine the percentage of used storage for my file system, excluding space reserved for cached writes on clients? | Total storage capacity utilization (percent) | `StorageCapacityUtilization` | | …determine the percentage of used storage for my file system, including space reserved for cached writes on clients? | Total storage capacity utilization (percent) | `StorageCapacityUtilizationWithCachedWrites` | | …determine the percentage of used storage for my file system’s OSTs excluding space reserved for cached writes on clients? | Total storage capacity utilization per OST (percent) | `StorageCapacityUtilization` | | …determine the percentage of used storage for my file system’s OSTs, including space reserved for cached writes on clients? | Total storage capacity utilization per OST with client grants (percent) | `StorageCapacityUtilizationWithCachedWrites` | | …determine my file system’s data compression ratio? | Compression savings | 100\$1(LogicalDiskUsage - PhysicalDiskUSage)/LogicalDiskUsage | **Object storage performance (in Performance tab)** | How do I... | Chart | Relevant metrics | | --- | --- | --- | | …determine the network throughput between the clients and the OSSs as a percentage of the provisioned limit? | Network throughput (percent) | NetworkThroughputUtilization | | …determine the disk throughput between the OSS and its OSTs as a percentage of the provisioned limit? | Disk throughput (percent) | `FileServerDiskThroughputUtilization` | | …determine the IOPS for operations that access OSTs as a percentage of the provisioned limit? | Disk IOPS (percent) | `DiskIopsUtilization` | **Metadata performance (in Performance tab)** | How do I... | Chart | Relevant metrics | | --- | --- | --- | | …determine the metadata server's CPU utilization percentage? | CPU utilization (percent) | CPUUtilization | | …determine the metadata IOPS utilization as a percentage of the provisioned limit? | Metadata IOPS utilization | MAX(MDT Disk IOPS) | # Accessing CloudWatch metrics You can access Amazon FSx for Lustre metrics for CloudWatch in the following ways: + The Amazon FSx for Lustre console. + The CloudWatch console. + The CloudWatch command line interface (CLI). + The CloudWatch API. The following procedures show you how to access the metrics using these tools. ## Using the Amazon FSx for Lustre console **To view metrics using the Amazon FSx for Lustre console** 1. Open the Amazon FSx console at [https://console.aws.amazon.com/fsx/](https://console.aws.amazon.com/fsx/). 1. From the navigation pane, choose **File systems**, then choose the file system that has the metrics that you want to view. 1. On the **Summary** page, choose **Monitoring & performance** to see the metrics for your file system. There are four tabs on the **Monitoring & performance** panel. + Choose **Summary** (the default tab) to display any active warnings, CloudWatch alarms, and graphs for **File system activity**. + Choose **Storage** to view storage capacity, utilization metrics, and active warnings. + Choose **Performance** to view file server and storage performance metrics, and active warnings. + Choose **CloudWatch alarms** to view graphs of any alarms configured for your file system. ## Using the CloudWatch console **To view metrics using the CloudWatch console** 1. Open the [CloudWatch console](https://console.aws.amazon.com/cloudwatch). 1. In the navigation pane, choose **Metrics**. 1. Select the **FSx** namespace. 1. (Optional) To view a metric, enter its name in the search field. 1. (Optional) To explore metrics, select the category that best matches your question. ## Using the AWS CLI **To access metrics from the AWS CLI** + Use the [https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/list-metrics.html](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/list-metrics.html) command with the `--namespace "AWS/FSx"` namespace. For more information, see the [AWS CLI Command Reference](https://docs.aws.amazon.com/cli/latest/reference/). ## Using the CloudWatch API **To access metrics from the CloudWatch API** + Call `[GetMetricStatistics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricStatistics.html)`. For more information, see [Amazon CloudWatch API Reference](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/). # Amazon FSx for Lustre metrics and dimensions Amazon FSx for Lustre publishes the metrics described in the following tables in the `AWS/FSx` namespace in Amazon CloudWatch for all FSx for Lustre file systems. **Topics** + [FSx for Lustre network I/O metrics](#fsx-networkio-metrics) + [FSx for Lustre object storage server metrics](#fsx-oss-metrics) + [FSx for Lustre object storage target metrics](#fsx-ost-metrics) + [FSx for Lustre metadata metrics](#fs-metadata-metrics) + [FSx for Lustre storage capacity metrics](#fsx-storage-capacity-metrics) + [FSx for Lustre S3 repository metrics](#auto-import-export-metrics) + [FSx for Lustre dimensions](#fsx-dimensions) ## FSx for Lustre network I/O metrics The `AWS/FSx` namespace includes the following network I/O metrics. All of these metrics take one dimension, `FileSystemId`. | Metric | Description | | --- | --- | | DataReadBytes | The number of bytes from reads by clients to the file system. The `Sum` statistic is the total number of bytes associated with read operations during the specified period. The `Minimum` statistic is the minimum number of bytes associated with read operations on a single OST. The `Maximum` statistic is the maximum number of bytes associated with read operations on the OST. The `Average` statistic is the average number of bytes associated with read operations per OST. The `SampleCount` statistic is the number of OSTs. To calculate the average throughput (bytes per second) for a period, divide the `Sum` statistic by the number of seconds in the period. Units: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/fsx/latest/LustreGuide/fs-metrics.html) Valid statistics: `Sum`, `Minimum`, `Maximum`, `Average`, `SampleCount` | | DataWriteBytes | The number of bytes from writes by clients to the file system. The `Sum` statistic is the total number of bytes associated with write operations. The `Minimum` statistic is the minimum number of bytes associated with write operations on a single OST. The `Maximum` statistic is the maximum number of bytes associated with write operations on the OST. The `Average` statistic is the average number of bytes associated with write operations per OST. The `SampleCount` statistic is the number of OSTs. To calculate the average throughput (bytes per second) for a period, divide the `Sum` statistic by the number of seconds in the period. Units: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/fsx/latest/LustreGuide/fs-metrics.html) Valid statistics: `Sum`, `Minimum`, `Maximum`, `Average`, `SampleCount` | | DataReadOperations | The number of read operations. The `Sum` statistic is the total number of read operations. The `Minimum` statistic is the minimum number of read operations on a single OST. The `Maximum` statistic is the maximum number of read operations on the OST. The `Average` statistic is the average number of read operations per OST. The `SampleCount` statistic is the number of OSTs. To calculate the average number of read operations (operations per second) for a period, divide the `Sum` statistic by the number of seconds in the period. Units: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/fsx/latest/LustreGuide/fs-metrics.html) Valid statistics: `Sum`, `Minimum`, `Maximum`, `Average`, `SampleCount` | | DataWriteOperations | The number of write operations. The `Sum` statistic is the total number of write operations. The `Minimum` statistic is the minimum number of write operations on a single OST. The `Maximum` statistic is the maximum number of write operations on the OST. The `Average` statistic is the average number of write operations per OST. The `SampleCount` statistic is the number of OSTs. To calculate the average number of write operations (operations per second) for a period, divide the `Sum` statistic by the number of seconds in the period. Units: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/fsx/latest/LustreGuide/fs-metrics.html) Valid statistics: `Sum`, `Minimum`, `Maximum`, `Average`, `SampleCount` | | MetadataOperations | The number of metadata operations. The `Sum` statistic is the count of metadata operations. The `Minimum` statistic is the minimum number of metadata operations per MDT. The `Maximum` statistic is the maximum number of metadata operations per MDT. The `Average` statistic is the average number of metadata operations per MDT. The `SampleCount` statistic is the number of MDTs. To calculate the average number of metadata operations (operations per second) for a period, divide the `Sum` statistic by the number of seconds in the period. Units: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/fsx/latest/LustreGuide/fs-metrics.html) Valid statistics: `Sum`, `Minimum`, `Maximum`, `Average`, `SampleCount` | | ClientConnections | The number of active connections between clients and the file system. Unit: Count | ## FSx for Lustre object storage server metrics The `AWS/FSx` namespace includes the following object storage server (OSS) metrics. All of these metrics take two dimensions, `FileSystemId` and `FileServer`. + `FileSystemId` – Your file system's AWS resource ID. + `FileServer` – The name of the object storage server (OSS) in your Lustre file system. Each OSS is provisioned with one or more object storage targets (OSTs). OSS use the naming convention of OSS, where *HostIndex* represents a 4-digit hexadecimal value (for example, `OSS0001`). An OSS's ID is the ID of the first OST attached to it. For example, the first OSS attached to `OST0000` and `OST0001`, will use `OSS0000`, and the second OSS attached to `OST0002`, `OST0003` will use `OSS0002`. | Metric | Description | | --- | --- | | NetworkThroughputUtilization | Network throughput utilization as a percentage of available network throughput for your file system. This metric is equivalent to the sum of `NetworkSentBytes` and `NetworkReceivedBytes` as a percentage of the network throughput capacity of one OSS for your file system. There is one metric emitted each minute for each of your file system's OSSs. The `Average` statistic is the average network throughput utilization for the given OSS over the specified period. The `Minimum` statistic is the lowest network throughput utilization for the given OSS over one minute, for the specified period. The `Maximum` statistic is the highest network throughput utilization for the given OSS over one minute, for the specified period. Unit: Percent Valid statistics: `Average`, `Minimum`, `Maximum` | | NetworkSentBytes | The number of bytes sent by the file system. All traffic is considered in this metric, including data movement to and from linked data repositories. There is one metric emitted each minute for each of your file system's OSSs. The `Sum` statistic is the total number of bytes sent over the network by the given OSS over the specified period. The `Average` statistic is the average number of bytes sent over the network by the given OSS over the specified period. The `Minimum` statistic is the lowest number of bytes sent over the network by the given OSS over the specified period. The `Maximum` statistic is the highest number of bytes sent over the network by the given OSS over the specified period. To calculate sent throughput (bytes per second) for any statistic, divide the statistic by the seconds in the specified period. Unit: Bytes Valid statistics: `Sum`, `Average`, `Minimum`, `Maximum` | | NetworkReceivedBytes | The number of bytes received by the file system. All traffic is considered in this metric, including data movement to and from linked data repositories. There is one metric emitted each minute for each of your file system's OSSs. The `Sum` statistic is the total number of bytes received over the network by the given OSS over the specified period. The `Average` statistic is the average number of bytes received over the network by the given OSS over the specified period. The `Minimum` statistic is the lowest number of bytes received over the network by the given OSS over the specified period. The `Maximum` statistic is the highest number of bytes received over the network by the given OSS over the specified period. To calculate throughput (bytes per second) for any statistic, divide the statistic by the seconds in the specified period. Unit: Bytes Valid statistics: `Sum`, `Average`, `Minimum`, `Maximum` | | FileServerDiskThroughputUtilization | The disk throughput between your OSS and associated OSTs, as a percentage of the provisioned limit determined by throughput capacity. This metric is equivalent to the sum of `DiskReadBytes` and `DiskWriteBytes` as a percentage of the OSS's disk throughput capacity for your file system. There is one metric emitted each minute for each of your file system's OSSs. The `Average` statistic is the average OSS disk throughput utilization for the given OSS over the specified period. The `Minimum`statistic is the lowest OSS disk throughput utilization for the given OSS over the specified period. The `Maximum` statistic is the highest OSS disk throughput utilization for the given OSS over the specified period. Unit: Percent Valid statistics: `Average`, `Minimum`, `Maximum` | ## FSx for Lustre object storage target metrics The `AWS/FSx` namespace includes the following object storage target (OST) metrics. All of these metrics take two dimensions, `FileSystemId` and `StorageTargetId`. **Note** `DiskReadOperations` and `DiskWriteOperations` metrics are not available on Scratch file systems, and `DiskIopsUtilization` metrics are not available on Scratch and Persistent HDD file systems. | Metric | Description | | --- | --- | | DiskReadBytes | The number of bytes (disk IO) from any disk reads from this OST. There is one metric emitted each minute for each of your file system's OSTs. The `Sum` statistic is the total number of bytes read in a minute from the given OST over the specified period. The `Average` statistic is the average number of bytes read each minute from the given OST over the specified period. The `Minimum` statistic is the lowest number of bytes read each minute from the given OST over the specified period. The `Maximum` statistic is the highest number of bytes read each minute from the given OST over the specified period. To calculate read disk throughput (bytes per second) for any statistic, divide the statistic by the seconds in the period. Unit: Bytes Valid statistics: `Sum`, `Average`, `Minimum`, and, `Maximum` | | DiskWriteBytes | The number of bytes (disk IO) from any disk writes from this OST. There is one metric emitted each minute for each of your file system's OSTs. The `Sum` statistic is the total number of bytes written each minute from the given OST over the specified period. The `Average` statistic is the average number of bytes written each minute from the given OST over the specified period. The `Minimum` statistic is the lowest number of bytes written each minute from the given OST over the specified period. The `Maximum` statistic is the highest number of bytes written each minute from the given OST over the specified period. To calculate read disk throughput (bytes per second) for any statistic, divide the statistic by the seconds in the period Unit: Bytes Valid statistics: `Sum`, `Average`, `Minimum`, and, `Maximum` | | DiskReadOperations | The number of read operations (disk IO) to this OST. There is one metric emitted each minute for each of your file system's OSTs. The `Sum` statistic is the total number of read operations performed by the given OST over the specified period. The `Average` statistic is the average number of read operations performed each minute by the given OST over the specified period. The `Minimum` statistic is the lowest number of read operations performed each minute by the given OST over the specified period. The `Maximum` statistic is the highest number of read operations performed each minute by the given OST over the specified period. To calculate average disk IOPS over the period, use the `Average` statistic and divide the result by 60 (seconds). Units: Count Valid statistics: `Sum`, `Average`, `Minimum`, and `Maximum` | | DiskWriteOperations | The number of write operations (disk IO) to this OST. There is one metric emitted each minute for each of your file system's OSTs. The `Sum` statistic is the total number of write operations performed by the given OST over the specified period. The `Average` statistic is the average number of write operations performed each minute by the given OST over the specified period. The `Minimum` statistic is the lowest number of write operations performed each minute by the given OST over the specified period. The `Maximum` statistic is the highest number of write operations performed each minute by the given OST over the specified period. To calculate average disk IOPS over the period, use the `Average` statistic and divide the result by 60 (seconds). Units: Count Valid statistics: `Sum`, `Average`, `Minimum`, and `Maximum` | | DiskIopsUtilization | The disk IOPS utilization of one OST, as a percentage of the OST's disk IOPS limit. There is one metric emitted each minute for each of your file system's OSTs. The `Average` statistic is the average disk IOPS utilization for the given OST over the specified period. The `Minimum` statistic is the lowest disk IOPS utilization for the given OST over the specified period. The `Maximum` statistic is the highest disk IOPS utilization for the given OST over the specified period. Unit: Percent Valid statistics: `Average`, `Minimum`, and `Maximum` | ## FSx for Lustre metadata metrics The `AWS/FSx` namespace includes the following metadata metrics. The `CPUUtilization` metric takes the `FileSystemId` and `FileServer` dimensions, while the other metrics take the `FileSystemId` and `StorageTargetId` dimensions. + `FileSystemId` – Your file system's AWS resource ID. + `StorageTargetId` – The name of the metadata target (MDT). MDTs use the naming convention of MDT (for example, `MDT0001`). + `FileServer` – The name of the metadata server (MDS) in your Lustre file system. Each MDS is provisioned with one metadata target (MDT). MDS use the naming convention of MDS, where `HostIndex` represents a 4-digit hexadecimal value derived using the MDT index on the server. For example, the first MDS provisioned with `MDT0000` will use `MDS0000`, and the second MDS provisioned with `MDT0001` will use `MDS0001`. Your file system contains multiple metadata servers if your file system has a metadata configuration specified. | Metric | Description | | --- | --- | | CPUUtilization | The percent utilization of your file system's MDS CPU resources. There is one metric emitted each minute for each of your file system's MDSs. The `Average` statistic is the average CPU utilization of the MDS over a specified period. The `Minimum` statistic is the lowest CPU utilization for the given MDS over the specified period. The `Maximum` statistic is the highest CPU utilization for the given MDS over the specified period. Unit: Percent Valid statistics: `Average`, `Minimum` and `Maximum` | | FileCreateOperations | Total number of file create operations. Unit: Count | | FileOpenOperations | Total number of file open operations. Unit: Count | | FileDeleteOperations | Total number of file delete operations. Unit: Count | | StatOperations | Total number of stat operations. Unit: Count | | RenameOperations | Total number of directory renames, whether in-place directory renames or cross directory renames. Unit: Count | ## FSx for Lustre storage capacity metrics The `AWS/FSx` namespace includes the following storage capacity metrics. All of these metrics take two dimensions, `FileSystemId` and `StorageTargetId` except `LogicalDiskUsage` and `PhysicalDiskUsage` which take the `FileSystemId` dimension. | Metric | Description | | --- | --- | | FreeDataStorageCapacity | The amount of available storage capacity in this OST. There is one metric emitted each minute for each of your file system's OSTs. The `Sum` statistic is the total number of bytes available in the given OST over the specified period. The `Average` statistic is the average number of bytes available in the given OST over the specified period. The `Minimum` statistic is the lowest number of bytes available in the given OST over the specified period. The `Maximum` statistic is the highest number of bytes available in the given OST over the specified period. Unit: Bytes Valid statistics: `Sum`, `Average`, `Minimum`, and `Maximum` | | `StorageCapacityUtilization` | The storage capacity utilization for a given file system OST. There is one metric emitted each minute for each of your file system's OSTs. The `Average` statistic is the average amount of storage capacity utilization for a given OST over a specified period. The `Minimum` statistic is the minimum amount of storage capacity utilization for a given OST over a specified period. The `Maximum` statistic is the maximum amount of storage capacity utilization for a given OST over a specified period. Unit: Percent Valid statistics: `Average`, `Minimum`, `Maximum` | | `StorageCapacityUtilizationWithCachedWrites` | The storage capacity utilization for a given file system OST including space reserved for cached writes on the client. There is one metric emitted each minute for each of your file system's OSTs. The `Average` statistic is the average amount of storage capacity utilization for a given OST over a specified period. The `Minimum` statistic is the minimum amount of storage capacity utilization for a given OST over a specified period. The `Maximum` statistic is the maximum amount of storage capacity utilization for a given OST over a specified period. Unit: Percent Valid statistics: `Average`, `Minimum`, `Maximum` | | LogicalDiskUsage | The amount of logical data stored (uncompressed). The `Sum` statistic is the total number of logical bytes stored in the file system. The `Minimum` statistic is the least number of logical bytes stored in an OST in the file system. The `Maximum` statistic is the largest number of logical bytes stored in an OST in the file system. The `Average` statistic is the average number of logical bytes stored per OST. The `SampleCount` statistic is the number of OSTs. Units: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/fsx/latest/LustreGuide/fs-metrics.html) Valid statistics: `Sum`, `Minimum`, `Maximum`, `Average`, `SampleCount` | | PhysicalDiskUsage | The amount of storage physically occupied by file system data (compressed). The `Sum` statistic is the total number of bytes occupied in OSTs in the file system. The `Minimum` statistic is the total number of bytes occupied in the emptiest OST. The `Maximum` statistic is the total number of bytes occupied in the fullest OST. The `Average` statistic is the average number of bytes occupied per OST. The `SampleCount` statistic is the number of OSTs. Units: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/fsx/latest/LustreGuide/fs-metrics.html) Valid statistics: `Sum`, `Minimum`, `Maximum`, `Average`, `SampleCount` | ## FSx for Lustre S3 repository metrics FSx for Lustre publishes the following `AutoImport` (automatic import) and `AutoExport` (automatic export) metrics into the `FSx` namespace in CloudWatch. These metrics use dimensions to enable more granular measurements of your data. All `AutoImport` and `AutoExport` metrics have the `FileSystemId` and `Publisher` dimensions. | Metric | Description | | --- | --- | | `AgeOfOldestQueuedMessage` Dimension: `AutoExport` | The age, in seconds, of the oldest message waiting to be exported. The `Average` statistic is the average age of the oldest message waiting to be exported. The `Maximum` statistic is the maximum number of seconds a message lived in the export queue. The `Minimum` statistic is the minimum number of seconds a message lived in the export queue. A value of zero indicates that no messages are waiting to be exported. Units: Seconds Valid statistics: `Average`, `Minimum`, `Maximum` | | `RepositoryRenameOperations` Dimension: `AutoExport` | The number of renames processed by the file system in response to a larger directory rename. The `Sum` statistic is the total number of rename operations that result from a directory rename. The `Average` statistic is the average number of rename operations for the file system. The `Maximum` statistic is the maximum number of rename operations associated with a directory rename on the file system. The `Minimum` statistic is the minimum number of renames associated with a directory rename on the file system. Units: Count Valid statistics: `Sum`, `Average`, `Minimum`, `Maximum`, | | `AgeOfOldestQueuedMessage` Dimension: `AutoImport` | The age, in seconds, of the oldest message waiting to be imported. The `Average` statistic is the average age of the oldest message waiting to be imported. The `Maximum` statistic is the maximum number of seconds a message lived in the import queue. The `Minimum` statistic is the minimum number of seconds a message lived in the import queue. A value of zero indicates that no messages are waiting to be imported. Units: Seconds Valid statistics: `Average`, `Minimum`, `Maximum` | ## FSx for Lustre dimensions Amazon FSx for Lustre metrics use the `AWS/FSx` namespace and use the following dimensions. + The `FileSystemId` dimension denotes a file system's ID and filters the metrics that you request to that individual file system. You can find the ID on the Amazon FSx console on the **Summary** panel of the file system details page, in the **File system ID** field. The file system ID takes the form of *fs-01234567890123456*. You can also see the ID in the response of a [https://docs.aws.amazon.com/cli/latest/reference/fsx/describe-file-systems.html](https://docs.aws.amazon.com/cli/latest/reference/fsx/describe-file-systems.html) CLI command (the equivalent API action is [https://docs.aws.amazon.com/fsx/latest/APIReference/API_DescribeFileSystems.html](https://docs.aws.amazon.com/fsx/latest/APIReference/API_DescribeFileSystems.html)). + The `StorageTargetId` dimension denotes which OST (object storage target) or MDT (metadata target) published the metadata metrics. A `StorageTargetId` takes the form of `OSTxxxx` (for example, `OST0001`) or `MDTxxxx` (for example, `MDT0001`). + The `FileServer` dimension denotes the following + For OSS metrics: the name of the object storage server (OSS). OSS use the `OSSxxxx` naming convention (for example, `OSS0002`). + For the CPUUtilization metric: the name of a metadata server (MDS). MDS use the `MDSxxxx` naming convention (for example, `MDS0002`). + The `Publisher` dimension is available in CloudWatch and AWS CLI for the `AutoImport` and `AutoImport` metrics to denote which service published the metrics. For more information about dimensions, see [Dimensions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html#Dimension) in the *Amazon CloudWatch User Guide*. # Performance warnings and recommendations FSx for Lustre displays a warning for CloudWatch metrics when one of these metrics approaches or crosses a predetermined threshold for multiple consecutive data points. These warnings provide you with actionable recommendations that you can use to optimize your file system's performance. Warnings are accessible in several areas of the **Monitoring & performance** dashboard on the Amazon FSx for Lustre console. All active or recent Amazon FSx performance warnings and CloudWatch alarms configured for the file system that are in an alarm state appear in the **Monitoring & performance** panel in the **Summary** section. The warning also appears in the section of the dashboard where the metric graph is displayed. These warnings automatically disappear from the dashboard 24 hours after the underlying metrics fall below the warning threshold. You can create CloudWatch alarms for any of the Amazon FSx metrics. For more information, see [Creating CloudWatch alarms to monitor metrics](creating_alarms.md). ## Use performance warnings to improve file system performance Amazon FSx provides actionable recommendations that you can use to optimize your file system's performance. You can take the recommended action if you expect the issue to continue, or if it's causing an impact to your file system's performance. Depending on which metric has triggered a warning, you can resolve it by increasing the file system's throughput capacity, storage capacity, or metadata IOPS, as described in the following table. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/fsx/latest/LustreGuide/performance-insights.html) For more information about file system performance, see [Amazon FSx for Lustre performance](performance.md). # Creating CloudWatch alarms to monitor metrics You can create a CloudWatch alarm that sends an Amazon SNS message when the alarm changes state. An alarm watches a single metric over a time period that you specify and performs one or more actions based on the value of the metric relative to a given threshold over a specified period of time. The action is a notification that's sent to an Amazon SNS topic or Auto Scaling policy. Alarms invoke actions for sustained state changes only. CloudWatch alarms don't invoke actions because they are in a particular state. The state must change and remain changed for a specified period of time. You can create an alarm on the Amazon FSx console or the CloudWatch console. The following procedures describe how to create alarms for Amazon FSx for Lustre using the console, AWS CLI, and API. ## To set alarms using the Amazon FSx for Lustre console 1. Open the Amazon FSx console at [https://console.aws.amazon.com/fsx/](https://console.aws.amazon.com/fsx/). 1. From the navigation pane, choose **File systems**, and then choose the file system that you want to create the alarm for. 1. On the **Summary** page, choose **Monitoring & performance**. 1. Choose **Create CloudWatch alarm**. You are redirected to the CloudWatch console. 1. Choose **Select metrics**, and choose **Next**. 1. In the **Metrics** section, choose **FSX**. 1. Choose **File System Metrics**, choose the metric that you want to set the alarm for, and then choose **Select metric**. 1. In the **Conditions** section, choose the conditions for the alarm, and choose **Next**. **Note** Metrics might not be published during file system maintenance. To prevent unnecessary and misleading alarm condition changes and to configure your alarms so that they are resilient to missing data points, see [Configuring how CloudWatch alarms treat missing data](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html#alarms-and-missing-data) in the *Amazon CloudWatch User Guide*. 1. If you want CloudWatch to send you an email or SNS notification when the alarm state triggers the action, choose **Whenever this alarm state is**. For **Select an SNS topic**, choose an existing SNS topic. If you select **Create topic**, you can set the name and email addresses for a new email subscription list. This list is saved and appears in the field for future alarms. Choose **Next**. **Warning** If you use **Create topic** to create a new Amazon SNS topic, the email addresses must be verified before they receive notifications. Emails are only sent when the alarm enters an alarm state. If this alarm state change happens before the email addresses are verified, they do not receive a notification. 1. Fill in the **Name**, **Description**, and **Whenever** values for the metric, and choose **Next**. 1. On the **Preview and create** page, review the alarm and choose **Create Alarm**. ## To set alarms using the CloudWatch console 1. Sign in to the AWS Management Console and open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. Choose **Create Alarm** to start the **Create Alarm Wizard**. 1. Choose **FSx Metrics** to locate a metric. To narrow the results, you can search for your file system ID. Select the metric that you want to create an alarm for and choose **Next**. 1. Enter a **Name** and a **Description**, and choose a **Whenever** value for the metric. 1. If you want CloudWatch to send you an email when the alarm state is reached, choose **State is ALARM** for **Whenever this alarm**. For **Send notification to**, choose an existing SNS topic. If you select **Create topic**, you can set up the names and email addresses for a new email subscription list. This list is saved and appears in the field for future alarms. **Warning** If you use **Create topic** to create a new Amazon SNS topic, the email addresses must be verified before they receive notifications. Emails are only sent when the alarm enters an alarm state. If this alarm state change happens before the email addresses are verified, they do not receive a notification. 1. View the **Alarm Preview** and then choose **Create Alarm** or go back to make changes. ## To set an alarm using the AWS CLI + Call `[put-metric-alarm](https://docs.aws.amazon.com/cli/latest/reference/put-metric-alarm.html)`. For more information, see *[AWS CLI Command Reference](https://docs.aws.amazon.com/cli/latest/reference/)*. ## To set an alarm using the CloudWatch + Call `[PutMetricAlarm](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_PutMetricAlarm.html)`. For more information, see *[Amazon CloudWatch API Reference](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/)*. # Logging with Amazon CloudWatch Logs FSx for Lustre supports logging of error and warning events for data repositories associated with your file system to Amazon CloudWatch Logs. **Note** Logging with Amazon CloudWatch Logs is only available on Amazon FSx for Lustre file systems created after 3pm PST on November 30, 2021. **Topics** + [Logging overview](#event-log-overview) + [Log destinations](#event-log-destinations) + [Managing logging](#manage-logging) + [Viewing logs](#view-logs) ## Logging overview If you have data repositories linked to your FSx for Lustre file system, you can enable logging of data repository events to Amazon CloudWatch Logs. Error and warning events can be logged for import, export, and restore events. For more information on these operations and on linking to data repositories, see [Using data repositories with Amazon FSx for Lustre](fsx-data-repositories.md). You can configure the log levels that Amazon FSx logs; that is, whether Amazon FSx will log only error events, only warning events, or both error and warning events. You can also turn event logging off at any time. **Note** We highly recommend that you enable logs for file systems that have any level of critical functionality associated with them. ## Log destinations When logging is enabled, FSx for Lustre must be configured with an Amazon CloudWatch Logs destination. The event log destination is an Amazon CloudWatch Logs log group, and Amazon FSx creates a log stream for your file system within this log group. CloudWatch Logs allows you to store, view, and search audit event logs in the Amazon CloudWatch console, run queries on the logs using CloudWatch Logs Insights, and trigger CloudWatch alarms or Lambda functions. You choose the log destination when you create your FSx for Lustre file system or afterwards by updating it. For more information, see [Managing logging](#manage-logging). By default, Amazon FSx will create and use a default CloudWatch Logs log group in your account as the event log destination. If you want to use a custom CloudWatch Logs log group as the event log destination, here are the requirements for the name and location of the event log destination: + The name of the CloudWatch Logs log group must begin with the `/aws/fsx/` prefix. + If you don't have an existing CloudWatch Logs log group when you create or update a file system on the console, Amazon FSx for Lustre can create and use a default log stream in the CloudWatch Logs `/aws/fsx/lustre` log group. The log stream will be created with the format `datarepo_file_system_id` (for example, `datarepo_fs-0123456789abcdef0`). + If you don't want to use the default log group, the configuration UI lets you create a CloudWatch Logs log group when you create or update your file system on the console. + The destination CloudWatch Logs log group must be in the same AWS partition, AWS Region, and AWS account as your Amazon FSx for Lustre file system. You can change the event log destination at any time. When you do so, new event logs are sent only to the new destination. ## Managing logging You can enable logging when you create a new FSx for Lustre file system or afterwards by updating it. Logging is turned on by default when you create a file system from the Amazon FSx console. However, logging is turned off by default when you create a file system with the AWS CLI or Amazon FSx API. On existing file systems that have logging enabled, you can change the event logging settings, including which log level to log events for, and the log destination. You can perform these tasks using the Amazon FSx console, AWS CLI, or Amazon FSx API. ### To enable logging when creating a file system (console) 1. Open the Amazon FSx console at [https://console.aws.amazon.com/fsx/](https://console.aws.amazon.com/fsx/). 1. Follow the procedure for creating a new file system described in [Step 1: Create your FSx for Lustre file system](getting-started.md#getting-started-step1) in the Getting Started section. 1. Open the **Logging - optional** section. Logging is enabled by default. ![\[\]](http://docs.aws.amazon.com/fsx/latest/LustreGuide/images/create-fs-logging.png) 1. Continue with the next section of the file system creation wizard. When the file system becomes **Available**, logging will be enabled. ### To enable logging when creating a file system (CLI) 1. When creating a new file system, use the `LogConfiguration` property with the [CreateFileSystem](https://docs.aws.amazon.com/fsx/latest/APIReference/API_CreateFileSystem.html) operation to enable logging for the new file system. ``` create-file-system --file-system-type LUSTRE \ --storage-capacity 1200 --subnet-id subnet-08b31917a72b548a9 \ --lustre-configuration "LogConfiguration={Level=WARN_ERROR, \ Destination="arn:aws:logs:us-east-1:234567890123:log-group:/aws/fsx/testEventLogging"}" ``` 1. When the file system becomes **Available**, logging feature will be enabled. ### To change the logging configuration (console) 1. Open the Amazon FSx console at [https://console.aws.amazon.com/fsx/](https://console.aws.amazon.com/fsx/). 1. Navigate to **File systems**, and choose the Lustre file system that you want to manage logging for. 1. Choose the **Data repository** tab. 1. On the **Logging** panel, choose **Update**. 1. On the **Update logging configuration** dialog, change the desired settings. 1. Choose **Log errors** to log only error events, or **Log warnings** to log only warning events, or both. Logging is disabled if you don't make a selection. 1. Choose an existing CloudWatch Logs log destination or create a new one. 1. Choose **Save**. ### To change the logging configuration (CLI) + Use the [https://docs.aws.amazon.com/cli/latest/reference/fsx/update-file-system.html](https://docs.aws.amazon.com/cli/latest/reference/fsx/update-file-system.html) CLI command or the equivalent [https://docs.aws.amazon.com/fsx/latest/APIReference/API_UpdateFileSystem.html](https://docs.aws.amazon.com/fsx/latest/APIReference/API_UpdateFileSystem.html) API operation. ``` update-file-system --file-system-id fs-0123456789abcdef0 \ --lustre-configuration "LogConfiguration={Level=WARN_ERROR, \ Destination="arn:aws:logs:us-east-1:234567890123:log-group:/aws/fsx/testEventLogging"}" ``` ## Viewing logs You can view the logs after Amazon FSx has started emitting them. You can view the logs as follows: + You can view logs by going to the Amazon CloudWatch console and choosing the log group and log stream to which your event logs are sent. For more information, see [ View log data sent to CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html) in the *Amazon CloudWatch Logs User Guide*. + You can use CloudWatch Logs Insights to interactively search and analyze your log data. For more information, see [ Analyzing log data with CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html), in the *Amazon CloudWatch Logs User Guide*. + You can also export logs to Amazon S3. For more information, see [ Exporting log data to Amazon S3](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3Export.html), in the *Amazon CloudWatch Logs User Guide*. To learn about failure reasons, see [Data repository event logs](data-repo-event-logs.md). # Logging FSx for Lustre API calls with AWS CloudTrail Amazon FSx for Lustre is integrated with AWS CloudTrail, a service that provides a record of actions taken by a user, role, or an AWS service in Amazon FSx for Lustre. CloudTrail captures all API calls for Amazon FSx for Lustre as events. Captured calls include calls from the Amazon FSx for Lustre console and from code calls to Amazon FSx for Lustre API operations. If you create a trail, you can enable continuous delivery of CloudTrail events to an Amazon S3 bucket, including events for Amazon FSx for Lustre. If you don't configure a trail, you can still view the most recent events in the CloudTrail console in **Event history**. Using the information collected by CloudTrail, you can determine the request that was made to Amazon FSx for Lustre. You can also determine the IP address from which the request was made, who made the request, when it was made, and additional details. To learn more about CloudTrail, see the [AWS CloudTrail User Guide](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/). ## Amazon FSx for Lustre information in CloudTrail CloudTrail is enabled on your AWS account when you create the account. When API activity occurs in Amazon FSx for Lustre, that activity is recorded in a CloudTrail event along with other AWS service events in **Event history**. You can view, search, and download recent events in your AWS account. For more information, see [Viewing Events with CloudTrail Event History](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events.html). For an ongoing record of events in your AWS account, including events for Amazon FSx for Lustre, create a trail. A *trail* enables CloudTrail to deliver log files to an Amazon S3 bucket. By default, when you create a trail in the console, the trail applies to all AWS Regions. The trail logs events from all AWS Regions in the AWS partition and delivers the log files to the Amazon S3 bucket that you specify. Additionally, you can configure other AWS services to further analyze and act upon the event data collected in CloudTrail logs. For more information, see the following topics in the *AWS CloudTrail User Guide:* + [Overview for Creating a Trail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-and-update-a-trail.html) + [CloudTrail Supported Services and Integrations](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-aws-service-specific-topics.html#cloudtrail-aws-service-specific-topics-integrations) + [Configuring Amazon SNS Notifications for CloudTrail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/getting_notifications_top_level.html) + [Receiving CloudTrail Log Files from Multiple Regions](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/receive-cloudtrail-log-files-from-multiple-regions.html) and [Receiving CloudTrail Log Files from Multiple Accounts](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-receive-logs-from-multiple-accounts.html) All Amazon FSx for Lustre [API calls](https://docs.aws.amazon.com/fsx/latest/APIReference/Welcome.html) are logged by CloudTrail. For example, calls to the `CreateFileSystem` and `TagResource` operations generate entries in the CloudTrail log files. Every event or log entry contains information about who generated the request. The identity information helps you determine the following: + Whether the request was made with root or AWS Identity and Access Management (IAM) user credentials. + Whether the request was made with temporary security credentials for a role or federated user. + Whether the request was made by another AWS service. For more information, see the [CloudTrail userIdentity Element](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-user-identity.html) in the *AWS CloudTrail User Guide.* ## Understanding Amazon FSx for Lustre log file entries A *trail* is a configuration that enables delivery of events as log files to an Amazon S3 bucket that you specify. CloudTrail log files contain one or more log entries. An *event* represents a single request from any source and includes information about the requested action, the date and time of the action, request parameters, and so on. CloudTrail log files aren't an ordered stack trace of the public API calls, so they don't appear in any specific order. The following example shows a CloudTrail log entry that demonstrates the `TagResource` operation when a tag for a file system is created from the console. ``` { "eventVersion": "1.05", "userIdentity": { "type": “Root”, "principalId": “111122223333”, "arn": "arn:aws:sts::111122223333:root”, "accountId": “111122223333”, "accessKeyId": "AKIAIOSFODNN7EXAMPLE”, "sessionContext": { "attributes": { "mfaAuthenticated": "false", "creationDate": "2018-11-14T22:36:07Z" } } }, "eventTime": "2018-11-14T22:36:07Z", "eventSource": "fsx.amazonaws.com", "eventName": "TagResource", "awsRegion": "us-east-1", "sourceIPAddress": “192.0.2.0”, "userAgent": “console.amazonaws.com”, "requestParameters": { "resourceARN": "arn:aws:fsx:us-east-1:111122223333:file-system/fs-ab12cd34ef56gh789” }, "responseElements": null, "requestID": “aEXAMPLE-abcd-1234-56ef-b4cEXAMPLE51”, "eventID": “bEXAMPLE-gl12-3f5h-3sh4-ab6EXAMPLE9p”, "eventType": "AwsApiCall", "apiVersion": "2018-03-01", "recipientAccountId": “111122223333” } ``` The following example shows a CloudTrail log entry that demonstrates the `UntagResource` action when a tag for a file system is deleted from the console. ``` { "eventVersion": "1.05", "userIdentity": { "type": “Root”, "principalId": "111122223333", "arn": "arn:aws:sts::111122223333:root", "accountId": "111122223333", "accessKeyId": "AKIAIOSFODNN7EXAMPLE", "sessionContext": { "attributes": { "mfaAuthenticated": "false", "creationDate": "2018-11-14T23:40:54Z" } } }, "eventTime": "2018-11-14T23:40:54Z", "eventSource": "fsx.amazonaws.com", "eventName": "UntagResource", "awsRegion": "us-east-1", "sourceIPAddress": "192.0.2.0", "userAgent": "console.amazonaws.com", "requestParameters": { "resourceARN": "arn:aws:fsx:us-east-1:111122223333:file-system/fs-ab12cd34ef56gh789" }, "responseElements": null, "requestID": "aEXAMPLE-abcd-1234-56ef-b4cEXAMPLE51", "eventID": "bEXAMPLE-gl12-3f5h-3sh4-ab6EXAMPLE9p", "eventType": "AwsApiCall", "apiVersion": "2018-03-01", "recipientAccountId": "111122223333" } ```