Automated deployment - Amazon S3 Glacier Re:Freezer

Automated deployment

Before you launch the solution, review the architecture, prerequisites, and other considerations discussed in this guide.

Follow the step-by-step instructions in this section to configure and deploy the solution.

Time to deploy: Approximately 10 minutes

Prerequisites

Important
  • Before deployment, ensure that there are no new uploads or deletes of archives occurring on your source Glacier vault. Your Glacier vault content must be static.

  • If you have archives that are larger than 5 TB each, contact your account team before running this deployment.

  1. Ensure you have reviewed the sections mentioned below before deployment.

  2. Record the name of the Glacier vault that you will use as the source location for the archives that you want to copy to the destination Amazon S3 bucket and S3 storage class. You will use this Glacier vault name during deployment.

  3. Create a new destination Amazon S3 bucket. This bucket will be the destination storage location for your Glacier vault archives. For more information, refer to Creating a bucket in the Amazon Simple Storage Service User Guide.

  4. Ensure that your account has permissions to deploy the CloudFormation template and create the necessary AWS IAM roles. Your account must have permissions to grant access to the source Glacier vault and destination Amazon S3 bucket.

Deployment overview

Use the following steps to deploy this solution on AWS. For detailed instructions, follow the links for each step. You must complete the above prerequisites before launching the stack.

Step 1. Launch the stack

  1. Launch the AWS CloudFormation template into your AWS account.

  2. Enter values for required parameters.

  3. Review the other template parameters, and adjust if necessary.

Step 2. Monitor the archive copy operation

  • Launch the CloudWatch dashboard and monitor the copy process.

Step 3. Post archive copy verification tasks

  • Verify that the archive copy operation was successful.

Step 1. Launch the stack

Important

This solution includes an option to send anonymous operational metrics to AWS. We use this data to better understand how customers use this solution and related services and products. AWS owns the data gathered though this survey. Data collection is subject to the AWS Privacy Policy.

To opt out of this feature, download the template, modify the AWS CloudFormation mapping section, and then use the AWS CloudFormation console to upload your template and deploy the solution. For more information, refer to the Collection of operational metrics section in this guide.

This automated AWS CloudFormation template deploys the Amazon S3 Glacier Re:Freezer solution in the AWS Cloud.

Note

You are responsible for the cost of the AWS services used while running this solution. For more details, refer to the Cost section in this guide, and refer to the pricing webpage for each AWS service used in this solution.

  1. Sign in to the AWS Management Console and use the button below to launch the amazon-s3-glacier-refreezer.template AWS CloudFormation template. 
                Amazon S3 Glacier Re:Freezer launch button
              Alternatively, you can download the template as a starting point for your own implementation.

  2. The template launches in the US East (N. Virginia) Region by default. To launch the solution in a different AWS Region, use the Region selector in the console navigation bar.

  3. On the Create stack page, verify that the correct template URL is in the Amazon S3 URL text box and select Next.

  4. On the Specify stack details page, assign a name to your solution stack. The stack name provided must be less than 12 characters. For information about naming character limitations, refer to IAM and STS Limits in the AWS Identity and Access Management User Guide.

  5. Under Parameters, review the parameters for this solution template and modify them as necessary. This solution uses the following default values.

Parameter Default Description
Glacier vault name <Requires input> Enter the source Glacier vault name that contains the archives to be copied (refer to prerequisite item #1).
S3 destination bucket name <Requires input> Enter the destination Amazon S3 bucket name for the copied archives (­prerequisite item #2).
S3 destination storage class Select value

Select the desired destination storage class for the restored content. (for example, Deep_Archive for Amazon S3 Glacier Deep Archive).

For more information refer to Amazon S3 Storage Classes.

Glacier retrieval option Select value

Select Bulk.

For more information, refer to Retrieving S3 Glacier Archives in the Amazon S3 Developer Guide.

Confirmation to avoid excessive costs Select value
Important

This solution deployment requires that you acknowledge both items in this parameter with YES to continue. If you select NO, the solution will NOT deploy.

  • If you are using CloudTrail on your destination Amazon S3 bucket, you must confirm that only a SINGLE CloudTrail export to S3 has been configured. More than one export could lead to excessive API charges.

  • If you have SNS notifications enabled on your Glacier vault, you must confirm that the default SNS notification topic for the source Glacier vault topic is deactivated. If it was not deactivated, it is set to receive restore notifications for ALL archives in the vault. If you have millions of archives to be restored this can lead to increased SNS communications to the SNS topic and to the receiving email channels.

[Optional] External filenames override for ArchiveDescription Blank

This is an optional field.

Use this option only if you want to customize the destination S3 object key-names to be different to the original names for the archive name set in the Glacier vaults ArchiveDescription value.

Enter the file path for the filename override using the following syntax: <S3-bucket-name>/<full-path-to-csv-file>

For more information, refer to the Translation of Glacier vault archive descriptions to S3 object names section.

  1. Choose Next.

  2. On the Configure stack options page, choose Next.

  3. On the Review page, review and confirm the settings. Check the box acknowledging that the template will create AWS Identity and Access Management (IAM) resources.

  4. Choose Create stack to deploy the stack.

    You can view the status of the stack in the AWS CloudFormation Console in the Status column. You should receive a CREATE_COMPLETE status in approximately 10 minutes.

Step 2. Monitor the archive copy operation

Select the Outputs tab, then choose the DashboardURL link to open your Amazon CloudWatch dashboard and monitor the progress of the archive copies from your Glacier vault to your Amazon S3 bucket.

When the Amazon S3 Glacier Re:Freezer solution is first deployed, it can take up to five hours for S3 Glacier to return the latest copy of the Glacier vault inventory report to the solution. The dashboard counters for Total Archives, Hashes Validated, and Copied to Destination start at zero, and take time before they start to update.

Figure 7: Sample CloudWatch dashboard

If the metrics shown in the Amazon S3 Glacier Re:Freezer Progress Metrics widget are rounded up (as shown in Figure 7), you can use the instructions below to change the widget settings to display more digits.

  1. From the Outputs tab, open the provided URL for your custom CloudWatch dashboard.

  2. On the top widget labeled Progress Metrics, select on the ellipsis and choose Edit.

    Figure 8: Edit in CloudWatch

  3. Select the Graph options tab.

  4. From the Number widget format section, check the box for Show as many digits as can fit, before rounding, then choose Update widget.

    The widget updates to display all digits for the counters.

    Figure 9: Updated widget with digits before rounding

Step 3. Post archive copy verification tasks

Use the following process to verify that the archive copy process has successfully copied all the archives from the source Glacier vault to the destination Amazon S3 bucket.

  1. Compare and validate that the value shown for the counter of Copied to Destination (from the CloudWatch Dashboard), and the total number of objects stored in your destination S3 bucket (S3 Console, CLI, API), matches the number of archives in your source Glacier vault (S3 Glacier console).

  • If these numbers match, then the process is complete and you can perform any additional post operations that you may require to view or access the S3 objects.

  • If the numbers do not match, then refer to the Troubleshooting topic before continuing to step 2 below.

  1. After you successfully verify that the source and target object counts match, and you do not need to keep the CloudWatch archive copy logs, you can delete the CloudFormation stack. For more information refer to the Uninstall the solution topic.