Automatically copy your Amazon S3 Glacier vault archives to an S3 bucket and storage classes - Data Transfer from Amazon S3 Glacier Vaults to Amazon S3

Automatically copy your Amazon S3 Glacier vault archives to an S3 bucket and storage classes

Publication date: December 2023 (last update: April 2024)

Data Transfer from Amazon S3 Glacier Vaults to Amazon S3 is a serverless solution that automates and optimizes the restore, copy, and transfer process of Amazon Simple Storage Service Glacier (Amazon S3 Glacier) vault archives. The solution copies all of the vault's archives to a defined Amazon Simple Storage Service (Amazon S3) bucket destination and storage class. Then you can attach tags to help you categorize your data, such as with data classification or cost allocation. A prebuilt Amazon CloudWatch dashboard provides a visualization of the copy operation progress.

Important

Amazon S3 and Amazon S3 Glacier are different AWS services.

Amazon S3 Glacier is an object storage service for low-cost data archiving and long-term backup. It stores archives in vaults. It doesn't offer storage classes. The Amazon S3 Glacier service provides a console. However, any archive operation, such as upload, download, or deletion, requires you to use the AWS CLI or write code. There is no console support for archive operations.

Amazon S3 is an object storage service for any type of data. It stores objects in buckets. It offers different storage classes for frequent access, infrequent access, archives, and optimized tiering. You can interact with the Amazon S3 service by using the Amazon S3 console or AWS Command Line Interface (AWS CLI).

The S3 Glacier Instant Retrieval, S3 Glacier Flexible Retrieval, and S3 Glacier Deep Archive storage classes are features of the Amazon S3 service. The S3 Glacier Flexible Retrieval storage class offers the same features as the Amazon S3 Glacier service. The Amazon S3 Glacier service doesn't offer storage classes.

For example, Saanvi works at AnyCompany Archives. Five years ago, she used the Amazon S3 Glacier service to store scanned copies of historical documents in a vault. AnyCompany just announced that they will have a different online exhibit each month, featuring documents that are stored in the S3 Glacier vault. To address this change of business:

  • Saanvi wants to take advantage of the storage classes offered with the Amazon S3 service, including more flexibility in how files are stored and accessed.

  • Using Data Transfer from Amazon S3 Glacier Vaults to Amazon S3, Saanvi can copy all of her document archives from her S3 Glacier vault to an S3 bucket. She can assign them to the S3 storage classes that best fit her use cases. For example, she can use the S3 Standard storage class for documents that will be featured in the first exhibit and accessed daily, and the S3 Glacier Deep Archive storage class for documents that won't be featured in any of the exhibits.

  • Now that the documents are stored in the Amazon S3 service, Saanvi can also apply S3 Lifecycle configurations, tag her data, and use the Amazon S3 console.

Note

This solution doesn't delete the original archives or the source S3 Glacier vault. You must manually delete the archives and vault. For more information, refer to Deleting an Archive in Amazon S3 Glacier in the Amazon S3 Glacier Developer Guide.

If your source S3 Glacier vault has a Vault Lock policy that prevents deletion, you must delete this policy before deleting the original archives. However, if your Vault Lock policy is in the Locked state, you can't delete it. See S3 Glacier Vault Lock and Abort Vault Lock (DELETE lock-policy) in the Amazon S3 Glacier Developer Guide for more information.

This implementation guide provides an overview of the Data Transfer from Amazon S3 Glacier Vaults to Amazon S3 solution, its reference architecture and components, considerations for planning the deployment, and configuration steps for deploying the solution to the Amazon Web Services (AWS) Cloud.

The intended audience for using this solution's features and capabilities in their environment includes solution architects, business decision makers, DevOps engineers, data scientists, and cloud professionals. Practical experience with the AWS Cloud, Amazon S3 Glacier vaults, Amazon S3 buckets, and Amazon S3 storage classes is preferred.

Use this navigation table to quickly find answers to these questions:

If you want to . . . Read . . .

Know the cost for running this solution.

The estimated cost for running this solution in the US East (Ohio) Region is USD $153.57 to copy 100,000 S3 Glacier vault archives, totaling 100 TB of data, from an S3 Glacier vault to an S3 bucket.

Cost
Understand the security considerations for this solution. Security

Know how to plan for quotas for this solution.

This solution uses AWS Lambda functions to transfer data. This affects your account-wide Lambda concurrency limit.

Quotas
Know which AWS Regions support this solution. Supported AWS Regions
View or download the AWS CloudFormation template included in this solution to automatically deploy the infrastructure resources (the "stack") for this solution. AWS CloudFormation template