Creating and updating Amazon S3 Storage Lens dashboards - Amazon Simple Storage Service
Creating an Amazon S3 Storage Lens dashboardUpdating an Amazon S3 Storage Lens dashboard

Creating and updating Amazon S3 Storage Lens dashboards

S3 Storage Lens aggregates your metrics and displays the information in the Account snapshot section on the Amazon S3 console Buckets page. S3 Storage Lens also provides an interactive dashboard that you can use to visualize insights and trends, flag outliers, and receive recommendations for optimizing storage costs and applying data-protection best practices. Your dashboard has drill-down options to generate and visualize insights at the organization, account, AWS Region, storage class, bucket, prefix, or Storage Lens group level. You can also send a daily metrics export in CSV or Parquet format to an S3 bucket.

The Amazon S3 Storage Lens default dashboard is default-account-dashboard. This dashboard is preconfigured by Amazon S3 to help you visualize summarized insights and trends for your entire account's aggregated free and advanced metrics on the console. You can't modify the default dashboard's configuration scope, but you can upgrade the metrics selection from the free metrics to the paid advanced metrics and recommendations, configure the optional metrics export, or even disable the default dashboard. The default dashboard cannot be deleted.

You can also create additional S3 Storage Lens custom dashboards that can be scoped to your organization in AWS Organizations or to specific Regions or buckets within an account.

Creating an Amazon S3 Storage Lens dashboard

Use the following steps to create an Amazon S3 Storage Lens dashboard on the Amazon S3 console.

Step 1: Define the dashboard scope

  1. Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.

  2. In the navigation bar on the top of the page, choose the name of the currently displayed AWS Region. Next, choose the Region that you want to switch to.

  3. In the left navigation pane, under S3 Storage Lens, choose Dashboards.

  4. Choose Create dashboard.

  5. On the Dashboard page, in the General section, do the following:

    1. View the Home Region for your dashboard. The home Region is the AWS Region where the configuration and metrics for this Storage Lens dashboard are stored.

    2. Enter a dashboard name.

      Dashboard names must be fewer than 65 characters and must not contain special characters or spaces.

      Note

      You can't change this dashboard name after the dashboard is created.

    3. You can optionally choose to add Tags to your dashboard. You can use tags to manage permissions for your dashboard and track costs for S3 Storage Lens.

      For more information, see Controlling access using resource tags in the IAM User Guide and AWS-Generated Cost Allocation Tags in the AWS Billing User Guide.

      Note

      You can add up to 50 tags to your dashboard configuration.

  6. In the Dashboard scope section, do the following:

    1. Choose the Regions and buckets that you want S3 Storage Lens to include or exclude in the dashboard.

    2. Choose the buckets in your selected Regions that you want S3 Storage Lens to include or exclude. You can either include or exclude buckets, but not both. This option is not available when you create organization-level dashboards.

    Note

    • You can either include or exclude Regions and buckets. This option is limited to Regions only when creating organization-level dashboards across member accounts in your organization.

    • You can choose up to 50 buckets to include or exclude.

Step 2: Configure the metrics selection

  1. In the Metrics selection section, choose the type of metrics that you want to aggregate for this dashboard.

    • To include free metrics aggregated at the bucket level and available for queries for 14 days, choose Free metrics.

    • To enable advanced metrics and other advanced options, choose Advanced metrics and recommendations. These options include advanced prefix aggregation, Amazon CloudWatch publishing, and contextual recommendations. Data is available for queries for 15 months. Advanced metrics and recommendations have an additional cost. For more information, see Amazon S3 pricing.

      For more information about advanced metrics and free metrics, see Metrics selection.

  2. Under Advanced metrics and recommendations features, select the options that you want to enable:

    • Advanced metrics

    • CloudWatch publishing

    • Prefix aggregation

    Important

    If you enable prefix aggregation for your S3 Storage Lens configuration, prefix-level metrics will not be published to CloudWatch. Only bucket, account, and organization-level S3 Storage Lens metrics are published to CloudWatch.

  3. If you enabled Advanced metrics, select the Advanced metrics categories that you want to display in your S3 Storage Lens dashboard:

    • Activity metrics

    • Detailed status code metrics

    • Advanced cost optimization metrics

    • Advanced data protection metrics

    For more information about metrics categories, see Metrics categories. For a complete list of metrics, see Amazon S3 Storage Lens metrics glossary.

  4. If you chose to enable prefix aggregation, configure the following:

    1. Choose the minimum prefix threshold size for this dashboard.

      For example, a prefix threshold of 5 percent indicates that prefixes that make up 5 percent or more of the bucket's total storage size will be aggregated.

    2. Choose the prefix depth.

      This setting indicates the maximum number of levels up to which the prefixes are evaluated. The prefix depth must be less than 10.

    3. Enter a prefix delimiter character.

      This value is used to identify each prefix level. The default value in Amazon S3 is the / character, but your storage structure might use other delimiter characters.

(Optional) Step 3: Export metrics for the dashboard

  1. In the Metrics export section, to create a metrics export that will be placed daily in a destination bucket of your choice, choose Enable.

    The metrics export is in CSV or Apache Parquet format. It represents the same scope of data as your S3 Storage Lens dashboard data without the recommendations.

  2. If you enabled the metrics export, choose the output format of your daily metrics export: CSV or Apache Parquet.

    Parquet is an open source file format for Hadoop that stores nested data in a flat columnar format.

  3. Choose the destination S3 bucket for your metrics export.

    You can choose a bucket in the current account of the S3 Storage Lens dashboard. Or you can choose another AWS account if you have the destination bucket permissions and the destination bucket owner's account ID.

  4. Choose the destination S3 bucket (format: s3://bucket-name/prefix).

    The bucket must be in the home Region of your S3 Storage Lens dashboard. The S3 console shows you the Destination bucket permission that will be added by Amazon S3 to the destination bucket policy. Amazon S3 updates the bucket policy on the destination bucket to allow S3 to place data in that bucket.

  5. (Optional) To enable server-side encryption for your metrics export, choose Specify an encryption key. Then, choose the Encryption type: Amazon S3 managed keys (SSE-S3) or AWS Key Management Service key (SSE-KMS).

    You can choose between an Amazon S3 managed key (SSE-S3) and an AWS Key Management Service (AWS KMS) key (SSE-KMS).

  6. (Optional) To specify an AWS KMS key, you must choose a KMS key or enter a key Amazon Resource Name (ARN).

    If you choose a customer managed key, you must grant S3 Storage Lens permission to encrypt in the AWS KMS key policy. For more information, see Using an AWS KMS key to encrypt your metrics exports.

  7. Choose Create dashboard.

To gain further visibility into your storage, you can create one or more S3 Storage Lens groups and attach them to your dashboard. An S3 Storage Lens group is a custom defined filter for objects based on prefixes, suffixes, object tags, object size, object age, or a combination of these filters.

You can use S3 Storage Lens groups to gain granular visibility into large shared buckets, such as data lakes, to make better-informed business decisions. For example, you can streamline storage allocation and optimize cost reporting by breaking down storage usage to specific groups of objects for individual projects and cost centers within a bucket or across multiple buckets.

To use S3 Storage Lens groups, you must upgrade your dashboard to use advanced metrics and recommendations. For more information about S3 Storage Lens groups, see Working with S3 Storage Lens groups.

Updating an Amazon S3 Storage Lens dashboard

Use the following steps to update an Amazon S3 Storage Lens dashboard on the Amazon S3 console.

Step 1: Update the dashboard scope

  1. Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.

  2. In the left navigation pane, choose Storage Lens, Dashboards.

  3. Choose the dashboard that you want to edit, and then choose Edit.

    The Edit dashboard page opens.

    Note

    You can't change the following:

    • The dashboard name

    • The home Region

    • The dashboard scope of the default dashboard, which is scoped to your entire account's storage

  4. (Optional) On the dashboard configuration page, in the General section, update and add tags to your dashboard.

    You can use tags to manage permissions for your dashboard and to track costs for S3 Storage Lens. For more information, see Controlling access using resource tags in the IAM User Guide and AWS-Generated Cost Allocation Tags in the AWS Billing User Guide.

    Note

    You can add up to 50 tags to your dashboard configuration.

  5. In the Dashboard scope section, do the following:

    1. Update the Regions and buckets that you want S3 Storage Lens to include or exclude in the dashboard.

      Note

      • You can either include or exclude Regions and buckets. This option is limited to Regions only when creating organization-level dashboards across member accounts in your organization.

      • You can choose up to 50 buckets to include or exclude.

    2. Update the buckets in your selected Regions that you want S3 Storage Lens to include or exclude. You can either include or exclude buckets, but not both. This option is not present when creating organization-level dashboards.

Step 2: Update the metrics selection

  1. In the Metrics selection section, choose the type of metrics that you want to aggregate for this dashboard.

    • To include free metrics aggregated at the bucket level and available for queries for 14 days, choose Free metrics.

    • To enable advanced metrics and other advanced options, choose Advanced metrics and recommendations. These options include advanced prefix aggregation, Amazon CloudWatch publishing, and contextual recommendations. Data is available for queries for 15 months. Advanced metrics and recommendations have an additional cost. For more information, see Amazon S3 pricing.

      For more information about advanced metrics and free metrics, see Metrics selection.

  2. Under Advanced metrics and recommendations features, select the options that you want to enable:

    • Advanced metrics

    • CloudWatch publishing

    • Prefix aggregation

    Important

    If you enable prefix aggregation for your S3 Storage Lens configuration, prefix-level metrics will not be published to CloudWatch. Only bucket, account, and organization-level S3 Storage Lens metrics are published to CloudWatch.

  3. If you enabled Advanced metrics, select the Advanced metrics categories that you want to display in your S3 Storage Lens dashboard:

    • Activity metrics

    • Detailed status code metrics

    • Advanced cost optimization metrics

    • Advanced data protection metrics

    For more information metrics categories, see Metrics categories. For a complete list of metrics, see Amazon S3 Storage Lens metrics glossary.

  4. If you chose to enable prefix aggregation, configure the following:

    1. Choose the minimum prefix threshold size for this dashboard.

      For example, a prefix threshold of 5 percent indicates that prefixes that make up 5 percent or more of the bucket's total storage size will be aggregated.

    2. Choose the prefix depth.

      This setting indicates the maximum number of levels up to which the prefixes are evaluated. The prefix depth must be less than 10.

    3. Enter a prefix delimiter character.

      This is the value used to identify each prefix level. The default value in Amazon S3 is the / character, but your storage structure might use other delimiter characters.

(Optional) Step 3: Export metrics for the dashboard

  1. In the Metrics export section, to create a metrics export that will be placed daily in a destination bucket of your choice, choose Enable. To disable the metrics export, choose Disable.

    The metrics export is in CSV or Apache Parquet format. It represents the same scope of data as your S3 Storage Lens dashboard data without the recommendations.

  2. If enabled, choose the output format of your daily metrics export: CSV or Apache Parquet.

    Parquet is an open source file format for Hadoop that stores nested data in a flat columnar format.

  3. Choose the destination S3 bucket for your metrics export.

    You can choose a bucket in the current account of the S3 Storage Lens dashboard. Or you can choose another AWS account if you have the destination bucket permissions and the destination bucket owner's account ID.

  4. Choose the destination S3 bucket (format: s3://bucket-name/prefix).

    The bucket must be in the home Region of your S3 Storage Lens dashboard. The S3 console shows you the Destination bucket permission that will be added by Amazon S3 to the destination bucket policy. Amazon S3 updates the bucket policy on the destination bucket to allow S3 to place data in that bucket.

  5. (Optional) To enable server-side encryption for your metrics export, choose Specify an encryption key. Then, choose the Encryption type: Amazon S3 managed keys (SSE-S3) or AWS Key Management Service key (SSE-KMS).

    You can choose between an Amazon S3 managed key (SSE-S3) and an AWS Key Management Service (AWS KMS) key (SSE-KMS).

  6. (Optional) To specify an AWS KMS key, you must choose a KMS key or enter a key Amazon Resource Name (ARN). Under AWS KMS key, specify your KMS key in one of the following ways:

    • To choose from a list of available KMS keys, choose Choose from your AWS KMS keys, and choose your KMS key from the list of available keys.

      Both the AWS managed key (aws/s3) and your customer managed keys appear in this list. For more information about customer managed keys, see Customer keys and AWS keys in the AWS Key Management Service Developer Guide.

      Note

      The AWS managed key (aws/S3) is not supported for SSE-KMS encryption with S3 Storage Lens.

    • To enter the KMS key ARN, choose Enter AWS KMS key ARN, and enter your KMS key ARN in the field that appears.

    • To create a new customer managed key in the AWS KMS console, choose Create a KMS key.

      If you choose a customer managed key, you must grant S3 Storage Lens permission to encrypt in the AWS KMS key policy. For more information, see Using an AWS KMS key to encrypt your metrics exports.

      For more information about creating an AWS KMS key, see Creating Keys in the AWS Key Management Service Developer Guide.

  7. Choose Save changes.

To gain further visibility into your storage, you can create one or more S3 Storage Lens groups and attach them to your dashboard. An S3 Storage Lens group is a custom defined filter for objects based on prefixes, suffixes, object tags, object size, object age, or a combination of these filters.

You can use S3 Storage Lens groups to gain granular visibility into large shared buckets, such as data lakes, to make better-informed business decisions. For example, you can streamline storage allocation and optimize cost reporting by breaking down storage usage to specific groups of objects for individual projects and cost centers within a bucket or across multiple buckets.

To use S3 Storage Lens groups, you must upgrade your dashboard to use advanced metrics and recommendations. For more information about S3 Storage Lens groups, see Working with S3 Storage Lens groups.