Tutorial: Getting started with S3 Express One Zone - Amazon Simple Storage Service

Tutorial: Getting started with S3 Express One Zone

Amazon S3 Express One Zone is the first S3 storage class where you can select a single Availability Zone with the option to co-locate your object storage with your compute resources which provides the highest possible access speed. Data in S3 Express One Zone is stored in S3 directory buckets. For more information on directory buckets, see Directory buckets.

S3 Express One Zone is ideal for any application where it's critical to minimize request latency. Such applications can be human-interactive workflows, like video editing, where creative professionals need responsive access to content from their user interfaces. S3 Express One Zone also benefits analytics and machine learning workloads that have similar responsiveness requirements from their data, especially workloads with a lot of smaller accesses or a large numbers of random accesses. S3 Express One Zone can be used with other AWS services such as Amazon EMR, Amazon Athena, AWS Glue Data Catalog and Amazon SageMaker Model Training to support analytics, artificial intelligence and machine learning (AI/ML) workloads,. You can work with the S3 Express One Zone storage class and directory buckets by using the Amazon S3 console, AWS SDKs, AWS Command Line Interface (AWS CLI), and Amazon S3 REST API. For more information, see What is S3 Express One Zone? and How is S3 Express One Zone different?.

This is an S3 Express One Zone workflow diagram.
Objective

In this tutorial, you will learn how to create a gateway endpoint, create and attach an IAM policy, create a directory bucket and then use the Import action to populate your directory bucket with objects currently stored in your general purpose bucket. Alternatively, you can manually upload objects to your directory bucket.

Prerequisites

Before you start this tutorial, you must have an AWS account that you can sign in to as an AWS Identity and Access Management (IAM) user with correct permissions.

Create an AWS account

To complete this tutorial, you need an AWS account. When you sign up for AWS, your AWS account is automatically signed up for all services in AWS, including Amazon S3. You are charged only for the services that you use. For more information about pricing, see S3 pricing.

Create an IAM user in your AWS account (console)

AWS Identity and Access Management (IAM) is an AWS service that helps administrators securely control access to AWS resources. IAM administrators control who can be authenticated (signed in) and authorized (have permissions) to access objects and use directory buckets in S3 Express One Zone. You can use IAM for no additional charge.

By default, users don't have permissions to access directory buckets and perform S3 Express One Zone operations. To grant access permissions for directory buckets and S3 Express One Zone operations, you can use IAM to create users or roles and attach permissions to those identities. For more information about how to create an IAM user, see Creating IAM users (console) in the IAM User Guide. For more information about how to create an IAM role, see Creating a role to delegate permissions to an IAM user in the IAM User Guide.

For simplicity, this tutorial creates and uses an IAM user. After completing this tutorial, remember to Delete the IAM user. For production use, we recommend that you follow the Security best practices in IAM in the IAM User Guide. A best practice requires human users to use federation with an identity provider to access AWS with temporary credentials. Another best practice is to require workloads to use temporary credentials with IAM roles to access AWS. To learn more about using AWS IAM Identity Center to create users with temporary credentials, see Getting started in the AWS IAM Identity Center User Guide.

Warning

IAM users have long-term credentials, which presents a security risk. To help mitigate this risk, we recommend that you provide these users with only the permissions they require to perform the task and that you remove these users when they are no longer needed.

Create an IAM policy and attach it to an IAM user or role (console)

By default, users don't have permissions for directory buckets and S3 Express One Zone operations. To grant access permissions for directory buckets, you can use IAM to create users, groups, or roles and attach permissions to those identities. Directory buckets are the only resource that you can include in bucket policies or IAM identity policies for S3 Express One Zone access.

To use Regional endpoint API operations (bucket-level or control plane operations) with S3 Express One Zone, you use the IAM authorization model, which doesn't involve session management. Permissions are granted for actions individually. To use Zonal endpoint API operations (object-level or data plane operations), you use CreateSession to create and manage sessions that are optimized for low-latency authorization of data requests. To retrieve and use a session token, you must allow the s3express:CreateSession action for your directory bucket in an identity-based policy or a bucket policy. If you're accessing S3 Express One Zone in the Amazon S3 console, through the AWS Command Line Interface (AWS CLI), or by using the AWS SDKs, S3 Express One Zone creates a session on your behalf. For more information, see CreateSession authorization and AWS Identity and Access Management (IAM) for S3 Express One Zone .

To create an IAM policy and attach the policy to an IAM user (or role)
  1. Sign in to the AWS Management Console and open the IAM Management Console.

  2. In the navigation pane, choose Policies.

  3. Choose Create Policy.

  4. Select JSON.

  5. Copy the policy below into the Policy editor window. Before you can create directory buckets or use S3 Express One Zone, you must grant the necessary permissions to your AWS Identity and Access Management (IAM) role or users. This example policy allows access to the CreateSession API operation (for use with other Zonal or object-level API operations) and all of the Regional endpoint (bucket-level) API operations. This policy allows the CreateSession API operation for use with all directory buckets, but the Regional endpoint API operations are allowed only for use with the specified directory bucket. To use this example policy, replace the user input placeholders with your own information.

    { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowAccessRegionalEndpointAPIs", "Effect": "Allow", "Action": [ "s3express:DeleteBucket", "s3express:DeleteBucketPolicy", "s3express:CreateBucket", "s3express:PutBucketPolicy", "s3express:GetBucketPolicy", "s3express:ListAllMyDirectoryBuckets" ], "Resource": "arn:aws:s3express:region:account_id:bucket/bucket-base-name--azid--x-s3/*" }, { "Sid": "AllowCreateSession", "Effect": "Allow", "Action": "s3express:CreateSession", "Resource": "*" } ] }
  6. Choose Next.

  7. Name the policy.

    Note

    Bucket tags are not supported for S3 Express One Zone.

  8. Select Create policy.

  9. Now that you've created an IAM policy, you can attach it to an IAM user. In the navigation pane, choose Policies.

  10. In the search bar, enter the name of your policy.

  11. From the Actions menu, select Attach.

  12. Under Filter by Entity Type, select IAM users or Roles.

  13. In the search field, type the name of the user or role you wish to use.

  14. Choose Attach Policy.

Step 1: Configure a gateway VPC endpoint

You can access both Zonal and Regional API operations through gateway virtual private cloud (VPC) endpoints. Gateway endpoints can allow traffic to reach S3 Express One Zone without traversing a NAT Gateway. We strongly recommend using gateway endpoints as they provide the most optimal networking path when working with S3 Express One Zone. You can access S3 Express One Zone directory buckets from your VPC without an internet gateway or NAT device for your VPC, and at no additional cost. Use the following procedure to configure a gateway endpoint that connects to S3 Express One Zone storage class objects and directory buckets.

To access S3 Express One Zone, you use Regional and Zonal endpoints that are different from standard Amazon S3 endpoints. Depending on the Amazon S3 API operation that you use, either a Zonal or Regional endpoint is required. For a complete list of supported API operations by endpoint type, see API operations supported by S3 Express One Zone . You must access both Zonal and Regional endpoints through a gateway virtual private cloud (VPC) endpoint.

Use the following procedure to create a gateway endpoint that connects to S3 Express One Zone storage class objects and directory buckets.

To configure a gateway VPC endpoint
  1. Open the Amazon VPC Console at https://console.aws.amazon.com/vpc/.

  2. In the side navigation pane under Virtual private cloud, choose Endpoints.

  3. Choose Create endpoint.

  4. Create a name for your endpoint.

  5. For Service category, choose AWS services.

  6. Under Services, search using the filter Type=Gateway and then choose the option button next to com.amazonaws.region.s3express.

  7. For VPC, choose the VPC in which to create the endpoint.

  8. For Route tables, select the route tables to be used by the endpoint. Amazon VPC automatically adds a route that points traffic destined for the service to the endpoint network interface.

  9. For Policy, choose Full access to allow all operations by all principals on all resources over the VPC endpoint. Otherwise, choose Custom to attach a VPC endpoint policy that controls the permissions that principals have to perform actions on resources over the VPC endpoint.

  10. Choose Create endpoint.

After creating a gateway endpoint, you can use Regional API endpoints and Zonal API endpoints to access Amazon S3 Express One Zone storage class objects and directory buckets.

Step 2: Create a directory bucket

  1. Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.

  2. In the navigation bar on the top of the page, choose the name of the currently displayed AWS Region. Next, choose the Region in which you want to create a bucket.

    Note

    To minimize latency and costs and address regulatory requirements, choose a Region close to you. Objects stored in a Region never leave that Region unless you explicitly transfer them to another Region. For a list of Amazon S3 AWS Regions, see AWS service endpoints in the Amazon Web Services General Reference.

  3. In the left navigation pane, choose Buckets.

  4. Choose Create bucket.

    The Create bucket page opens.

  5. Under General configuration, view the AWS Region where your bucket will be created.

  6. Under Bucket type, choose Directory.

    Note
    • If you've chosen a Region that doesn't support directory buckets, the Bucket type option disappears, and the bucket type defaults to a general purpose bucket. To create a directory bucket, you must choose a supported Region. For a list of Regions that support directory buckets and the Amazon S3 Express One Zone storage class, see S3 Express One Zone Availability Zones and Regions.

    • After you create the bucket, you can't change the bucket type.

    For Availability Zone, choose a Availability Zone local to your compute services. For a list of Availability Zones that support directory buckets and the S3 Express One Zone storage class, see S3 Express One Zone Availability Zones and Regions.

    Note

    The Availability Zone can't be changed after the bucket is created.

  7. Under Availability Zone , select the check box to acknowledge that in the event of an Availability Zone outage, your data might be unavailable or lost.

    Important

    Although directory buckets are stored across multiple devices within a single Availability Zone, directory buckets don't store data redundantly across Availability Zones.

  8. For Bucket name, enter a name for your directory bucket.

    Directory bucket names must:

    • Be unique within the chosen AWS Region and Availability Zone.

    • Name must be between 3 (min) and 63 (max) characters long, including the suffix.

    • Consists only of lowercase letters, numbers and hyphens (-).

    • Begin and end with a letter or number.

    • Must include the following suffix: --azid--x-s3.

    A suffix is automatically added to the base name that you provide when you create a directory bucket using the console. This suffix includes the Availability Zone ID of the Availability Zone that you chose.

    After you create the bucket, you can't change its name. For more information about naming buckets, see Bucket naming rules.

    Important

    Do not include sensitive information, such as account numbers, in the bucket name. The bucket name is visible in the URLs that point to the objects in the bucket.

  9. Under Object Ownership, the Bucket owner enforced setting is automatically enabled, and all access control lists (ACLs) are disabled. For directory buckets, ACLs can't be enabled.

    ACLs disabled
    • Bucket owner enforced (default) – ACLs are disabled, and the bucket owner automatically owns and has full control over every object in the bucket. ACLs no longer affect access permissions to data in the S3 bucket. The bucket uses policies exclusively to define access control.

      A majority of modern use cases in Amazon S3 no longer require the use of ACLs. For more information, see Controlling ownership of objects and disabling ACLs for your bucket.

  10. Under Block Public Access settings for this bucket, all Block Public Access settings for your directory bucket are automatically enabled. These settings can't be modified for directory buckets. For more information about blocking public access, see Blocking public access to your Amazon S3 storage.

  11. Under Server-side encryption settings, Amazon S3 applies server-side encryption with Amazon S3 managed keys (SSE-S3) as the base level of encryption for all S3 buckets. All object uploads to directory buckets are encrypted with SSE-S3. For directory buckets, the encryption type can't be modified. For more information about SSE-S3, see Using server-side encryption with Amazon S3 managed keys (SSE-S3).

  12. Choose Create bucket.

    After creating the bucket, you can add files and folders to the bucket. For more information, see Working with objects in a directory bucket.

The following step demonstrates how to use the Import action in the Amazon S3 console to populate your directory bucket with data.

Step 3: Importing data into a directory bucket

To complete this step, you must have a general purpose bucket that contains objects and is located in the same AWS Region as your directory bucket.

After you create a directory bucket in Amazon S3, you can populate the new bucket with data by using the Import action in the Amazon S3 console. Import simplifies copying data into directory buckets by letting you choose a prefix or a general purpose bucket to Import data from without having to specify all of the objects to copy individually. Import uses S3 Batch Operations which copies the objects in the selected prefix or general purpose bucket. You can monitor the progress of the Import copy job through the S3 Batch Operations job details page.

To use the Import action
  1. Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.

  2. In the navigation bar on the top of the page, choose the name of the currently displayed AWS Region. Next, choose the Region associated with the Availability Zone in which your directory bucket is located.

  3. In the left navigation pane, choose Buckets, and then choose the Directory buckets tab. Choose the directory bucket that you want to import objects into.

  4. Choose Import.

  5. For Source, enter the general purpose bucket (or bucket path including prefix) that contains the objects that you want to import. To choose an existing general purpose bucket from a list, choose Browse S3.

  6. In the Permissions section, you can choose to have an IAM role auto-generated. Alternatively, you can select an IAM role from a list, or directly enter an IAM role ARN.

    • To allow Amazon S3 to create a new IAM role on your behalf, choose Create new IAM role.

      Note

      If your source objects are encrypted with server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS), don't choose the Create new IAM role option. Instead, specify an existing IAM role that has the kms:Decrypt permission.

      Amazon S3 will use this permission to decrypt your objects. During the import process, Amazon S3 will then re-encrypt those objects by using server-side encryption with Amazon S3 managed keys (SSE-S3).

    • To choose an existing IAM role from a list, choose Choose from existing IAM roles.

    • To specify an existing IAM role by entering its Amazon Resource Name (ARN), choose Enter IAM role ARN, then enter the ARN in the corresponding field.

  7. Review the information that's displayed in the Destination and Copied object settings sections. If the information in the Destination section is correct, choose Import to start the copy job.

    The Amazon S3 console displays the status of your new job on the Batch Operations page. For more information about the job, choose the option button next to the job name, and then on the Actions menu, choose View details. To open the directory bucket that the objects will be imported into, choose View import destination.

Step 4: Manually upload objects to your directory bucket

You can also manually upload objects to your directory bucket.

To manually upload objects
  1. Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.

  2. In the navigation bar on the upper right corner of the page, choose the name of the currently displayed AWS Region. Next, choose the Region associated with the Availability Zone in which your directory bucket is located.

  3. In the left navigation pane, choose Buckets.

  4. Choose the Directory buckets tab.

  5. Choose the name of the bucket that you want to upload your folders or files to.

    Note

    If you chose the same directory bucket that you used in previous steps of this tutorial, your directory bucket will contain the objects that were uploaded from the Import tool. Notice that these objects are now stored in the S3 Express One Zone storage class.

  6. In the Objects list, choose Upload.

  7. On the Upload page, do one of the following:

    • Drag and drop files and folders to the dotted upload area.

    • Choose Add files or Add folder, choose the files or folders to upload, and then choose Open or Upload.

  8. Under Checksums, choose the Checksum function that you want to use.

    Note

    We recommend using CRC32 and CRC32C for the best performance with the S3 Express One Zone storage class. For more information, see S3 additional checksum best practices.

    (Optional) If you're uploading a single object that's less than 16 MB in size, you can also specify a pre-calculated checksum value. When you provide a pre-calculated value, Amazon S3 compares it with the value that it calculates by using the selected checksum function. If the values don't match, the upload won't start.

  9. The options in the Permissions and Properties sections are automatically set to default settings and can't be modified. Block Public Access is automatically enabled, and S3 Versioning and S3 Object Lock can't be enabled for directory buckets.

    (Optional) If you want to add metadata in key-value pairs to your objects, expand the Properties section, and then in the Metadata section, choose Add metadata.

  10. To upload the listed files and folders, choose Upload.

    Amazon S3 uploads your objects and folders. When the upload is finished, you see a success message on the Upload: status page.

    You have successfully created a directory bucket and uploaded objects to your bucket.

Step 5: Empty your directory bucket

You can empty your Amazon S3 directory bucket by using the Amazon S3 console.

To empty a directory bucket
  1. Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.

  2. In the navigation bar on the upper right corner of the page, choose the name of the currently displayed AWS Region. Next, choose the Region associated with the Availability Zone in which your directory bucket is located.

  3. In the left navigation pane, choose Buckets.

  4. Choose the Directory buckets tab.

  5. Choose the option button next to the name of the bucket that you want to empty, and then choose Empty.

  6. On the Empty bucket page, confirm that you want to empty the bucket by entering permanently delete in the text field, and then choose Empty.

  7. Monitor the progress of the bucket emptying process on the Empty bucket: status page.

Step 6: Delete your directory bucket

After you empty your directory bucket and abort all in-progress multipart uploads, you can delete your bucket by using the Amazon S3 console.

To delete a directory bucket
  1. Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.

  2. In the navigation bar on the upper right corner of the page, choose the name of the currently displayed AWS Region. Next, choose the Region associated with the Availability Zone in which your directory bucket is located.

  3. In the left navigation pane, choose Buckets.

  4. Choose the Directory buckets tab.

  5. In the Directory buckets list, choose the option button next to the bucket that you want to delete.

  6. Choose Delete.

  7. On the Delete bucket page, enter the name of the bucket in the text field to confirm the deletion of your bucket.

    Important

    Deleting a directory bucket can't be undone.

  8. To delete your directory bucket, choose Delete bucket.

Next steps

In this tutorial, you have learned how to create a directory bucket and use the S3 Express One Zone storage class. After completing this tutorial, you can explore related AWS services to use with the S3 Express One Zone storage class.

You can use the following AWS services with the S3 Express One Zone storage class to support your specific low-latency use case.

  • Amazon Elastic Compute Cloud (Amazon EC2) – Amazon EC2 provides secure and scalable computing capacity in the AWS Cloud. Using Amazon EC2 lessens your need to invest in hardware up front, so you can develop and deploy applications faster. You can use Amazon EC2 to launch as many or as few virtual servers as you need, configure security and networking, and manage storage.

  • AWS Lambda – Lambda is a compute service that lets you run code without provisioning or managing servers. You configure notification settings on a bucket, and grant Amazon S3 permission to invoke a function on the function's resource-based permissions policy.

  • Amazon Elastic Kubernetes Service (Amazon EKS) – Amazon EKS is a managed service that eliminates the need to install, operate, and maintain your own Kubernetes control plane on AWS. Kubernetes is an open-source system that automates the management, scaling, and deployment of containerized applications.

  • Amazon Elastic Container Service (Amazon ECS) – Amazon ECS is a fully managed container orchestration service that helps you easily deploy, manage, and scale containerized applications.

  • Amazon EMR – Amazon EMR is a managed cluster platform that simplifies running big data frameworks, such as Apache Hadoop and Apache Spark on AWS to process and analyze vast amounts of data.

  • Amazon Athena – Athena is an interactive query service that makes it easy to analyze data directly in Amazon S3 by using standard SQL. You can also use Athena to interactively run data analytics by using Apache Spark without having to plan for, configure, or manage resources. When you run Apache Spark applications on Athena, you submit Spark code for processing and receive the results directly.

  • AWS Glue Data Catalog – AWS Glue is a serverless data-integration service that makes it easy for analytics users to discover, prepare, move, and integrate data from multiple sources. You can use AWS Glue for analytics, machine learning, and application development. AWS Glue Data Catalog is a centralized repository that stores metadata about your organization's data sets. It acts as an index to the location, schema, and run-time metrics of your data sources.

  • Amazon SageMaker Runtime Model Training – Amazon SageMaker Runtime is a fully managed machine learning service. With SageMaker Runtime, data scientists and developers can quickly and easily build and train machine learning models, and then directly deploy them into a production-ready hosted environment.

For more information on S3 Express One Zone, see What is S3 Express One Zone? and How is S3 Express One Zone different?.