Uploading Large Archives in Parts (Multipart Upload)
Multipart Upload Process
As described in Uploading an Archive in Amazon Glacier, we encourage Amazon Glacier customers to use Multipart Upload to upload archives greater than 100 MB.
Initiate Multipart Upload
When you send a request to initiate a multipart upload, Amazon Glacier returns a multipart upload ID, which is a unique identifier for your multipart upload. Any subsequent multipart upload operations require this ID. The ID is valid for at least 24 hours.
In your request to initiate a multipart upload, you must specify the part size in number of bytes. Each part you upload, except the last part, must be of this size.
You don't need to know the overall archive size when using the Multipart Upload. This allows for use cases where the archive size is not known when you start uploading the archive. You only need to decide part size at the time you initiate a multipart upload.
In the initiate multipart upload request, you can also provide an optional archive description.
For each part upload request, you must include the multipart upload ID you obtained in step 1. In the request, you must also specify the content range, in bytes, identifying the position of the part in the final archive. Amazon Glacier later uses the content range information to assemble the archive in proper sequence. Because you provide the content range for each part that you upload, it determines the part's position in the final assembly of the archive, and therefore you can upload parts in any order. You can also upload parts in parallel. If you upload a new part using the same content range as a previously uploaded part, the previously uploaded part is overwritten.
Complete (or Abort) Multipart Upload
After uploading all the archive parts, you use the complete operation. Again, you must specify the upload ID in your request. Amazon Glacier creates an archive by concatenating parts in ascending order based on the content range you provided. Amazon Glacier response to a Complete Multipart Upload request includes an archive ID for the newly created archive. If you provided an optional archive description in the Initiate Multipart Upload request, Amazon Glacier associates it with assembled archive. After you successfully complete a multipart upload, you cannot refer to the multipart upload ID. That means you cannot access parts associated with the multipart upload ID.
If you abort a multipart upload, you cannot upload any more parts using that multipart upload ID. All storage consumed by any parts associated with the aborted multipart upload is freed. If any part uploads were in-progress, they can still succeed or fail even after you abort.
Additional Multipart Upload Operations
Amazon Glacier provides the following additional multipart upload API calls.
List Parts—Using this operation, you can list the parts of a specific multipart upload. It returns information about the parts that you have uploaded for a multipart upload. For each list parts request, Amazon Glacier returns information for up to 1,000 parts. If there are more parts to list for the multipart upload, the result is paginated and a marker is returned in the response at which to continue the list. You need to send additional requests to retrieve subsequent parts. Note that the returned list of parts doesn't include parts that haven't completed uploading.
List Multipart Uploads—Using this operation, you can obtain a list of multipart uploads in progress. An in-progress multipart upload is an upload that you have initiated, but have not yet completed or aborted. For each list multipart uploads request, Amazon Glacier returns up to 1,000 multipart uploads. If there are more multipart uploads to list, then the result is paginated and a marker is returned in the response at which to continue the list. You need to send additional requests to retrieve the remaining multipart uploads.
The following table provides multipart upload core specifications.
|Maximum archive size||10,000 x 4 GB|
|Maximum number of parts per upload||10,000|
1 MB to 4 GB, last part can be < 1 MB. You specify the size value in bytes.
The part size must be a megabyte (1024 KB) multiplied by a power of 2. For example,
|Maximum number of parts returned for a list parts request||1,000|
|Maximum number of multipart uploads returned in a list multipart uploads request||1,000|