Menu
Amazon Simple Storage Service
API Reference (API Version 2006-03-01)

Signature Calculations for the Authorization Header: Transferring Payload in Multiple Chunks (Chunked Upload) (AWS Signature Version 4)

As described in the Overview, when authenticating requests using the Authorization header, you have an option of uploading the payload in chunks. You can send data in fixed size or variable size chunks. This section describes the signature calculation process in chunked upload, how you create the chunk body, and how the delayed signing works where you first upload the chunk, and send its signature in the subsequent chunk. The example section (see Example: PUT Object) shows signature calculations and resulting Authorization headers that you can use as test suite to verify your code.

Note

When transferring data in a series of chunks, you must use the Content-Length HTTP header to explicitly specify the total content length (object length in bytes plus metadata in each chunk). This will require you to pre-compute the total length of the payload including the metadata you will send in each chunk before starting your request. The x-amz-decoded-content-length header will contain the size of the object length in bytes.

Each chunk signature calculation includes the signature of the previous chunk. To begin with, you create a seed signature using only the headers. You use the seed signature in the signature calculation of the first chunk. For each subsequent chunk, you create a chunk signature that includes signature of the previous chunk. Thus, the chunk signatures are chained together; that is, signature of chunk n is a function F(chunk n, signature(chunk n-1)). The chaining ensures you send the chunks in correct order.

To perform a chunked upload, do the following:

  1. Decide payload chunk size. You need this when you write the code.

    Chunk size must be at least 8 KB. We recommend a chunk size of a least 64 KB for better performance. This chunk size applies to all chunk except the last one. The last chunk you send can be smaller than 8 KB. If your payload is small and can fit in one chunk, then it can be smaller than the 8 KB.

  2. Create the seed signature for inclusion in the first chunk. For more information, see Calculating the Seed Signature.

  3. Create the first chunk and stream it. For more information, see Defining the Chunk Body.

  4. For each subsequent chunk, calculate the chunk signature that includes the previous signature in the string you sign, construct the chunk and send it. For more information, see Defining the Chunk Body.

  5. Send the final additional chunk, same as other chunks in construction, but it has zero data bytes. For more information, see Defining the Chunk Body.

Calculating the Seed Signature

The following diagram illustrates the process of calculating the seed signature.

The following table describes the functions that are shown in the diagram. You need to implement code for these functions.

Function Description
Lowercase() Convert the string to lowercase.
Hex() Lowercase base 16 encoding.
SHA256Hash() Secure Hash Algorithm (SHA) cryptographic hash function.
HMAC-SHA256() Computes HMAC by using the SHA256 algorithm with the signing key provided. This is the final signature.
Trim() Remove any leading or trailing whitespace.
UriEncode()

URI encode every byte. UriEncode() must enforce the following rules:

  • URI encode every byte except the unreserved characters: 'A'-'Z', 'a'-'z', '0'-'9', '-', '.', '_', and '~'.

  • The space character is a reserved character and must be encoded as "%20" (and not as "+").

  • Each URI encoded byte is formed by a '%' and the two-digit hexadecimal value of the byte.

  • Letters in the hexadecimal value must be uppercase, for example "%1A".

  • Encode the forward slash character, '/', everywhere except in the object key name. For example, if the object key name is photos/Jan/sample.jpg, the forward slash in the key name is not encoded.

Important

The standard UriEncode functions provided by your development platform may not work because of differences in implementation and related ambiguity in the underlying RFCs. We recommend that you write your own custom UriEncode function to ensure that your encoding will work.

The following is an example UriEncode() function in Java.

Copy
public static String UriEncode(CharSequence input, boolean encodeSlash) { StringBuilder result = new StringBuilder(); for (int i = 0; i < input.length(); i++) { char ch = input.charAt(i); if ((ch >= 'A' && ch <= 'Z') || (ch >= 'a' && ch <= 'z') || (ch >= '0' && ch <= '9') || ch == '_' || ch == '-' || ch == '~' || ch == '.') { result.append(ch); } else if (ch == '/') { result.append(encodeSlash ? "%2F" : ch); } else { result.append(toHexUTF8(ch)); } } return result.toString(); }

For information about the signing process, see Signature Calculations for the Authorization Header: Transferring Payload in a Single Chunk (AWS Signature Version 4). The process is the same except that the creation of CanonicalRequest differs as follows:

  • In addition to the request headers you plan to add, you must include the following headers:

    Header Description
    x-amz-content-sha256

    This header is required for all AWS Signature Version 4 requests. Set the value to STREAMING-AWS4-HMAC-SHA256-PAYLOAD to indicate that the signature covers only headers and that there is no payload.

    Content-Encoding

    Set the value to aws-chunked.

    Amazon S3 supports multiple content encodings. For example:

    Copy
    Content-Encoding : aws-chunked,gzip

    That is, you can specify your custom content-encoding when using Signature Version 4 streaming API.

    Note

    S3 will store the resulting object without the aws-chunked encoding. Therefore, when you retrieve the object it will not be aws-chunked encoded.

    x-amz-decoded-content-length Set the value to the length, in bytes, of the data to be chunked, without counting any metadata. For example, if you are uploading a 4 GB file, set the value to 4294967296. This is the raw size of the object to be uploaded (data you want to store in S3).
    Content-Length

    Set the value to the actual size of the transmitted HTTP body, which includes the length of your data (value set for x-amz-decoded-content-length) plus, chunk metadata. Each chunk will have metadata, such as the signature of the previous chunk. Chunk calculations are discussed in the following section.

You send the first chunk with the seed signature. You will need to construct the chunk as described in the following section.

Defining the Chunk Body

All chunks include some metadata. Each chunk must conform to the following structure:

Copy
string(IntHexBase(chunk-size)) + ";chunk-signature=" + signature + \r\n + chunk-data + \r\n

Where:

  • IntHexBase() is a function that you will write to convert an integer chunk-size to hexadecimal. For example, if chunk-size is 65536, hexadecimal string is "1000".

  • chunk-size is the size, in bytes, of the chunk-data, without metadata. For example, if you are uploading a 65 KB object and using a chunk size of 64 KB, you upload the data in three chunks: the first would be 64 KB, the second 1 KB, and the final chunk with 0 bytes.

  • signature For each chunk, you calculate signature using the following string to sign. For the first chunk, you use the seed-signature as the previous signature.

The size of the final chunk data that you send is 0, although the chunk body will still contain metadata, including the signature of the previous chunk.

Example: PUT Object

You can use the examples in this section as a reference to check signature calculations in your code. Before you review the examples, note the following:

  • The signature calculations in these examples use the following example security credentials.

    Parameter Value
    AWSAccessKeyId AKIAIOSFODNN7EXAMPLE
    AWSSecretAccessKey wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
  • All examples use the request timestamp 20130524T000000Z (Fri, 24 May 2013 00:00:00 GMT).

  • All examples use examplebucket as the bucket name.

  • The bucket is assumed to be in the US East (N. Virginia) region, and the credential Scope and the Signing Key calculations use us-east-1 as the region specifier. For more information, see Regions and Endpoints in the Amazon Web Services General Reference.

  • You can use either path style or virtual-hosted style requests. The examples below show use virtual-hosted style requests, for example:

    Copy
    https://examplebucket.s3.amazonaws.com/photos/photo1.jpg

    For more information, see Virtual Hosting of Buckets in the Amazon Simple Storage Service Developer Guide.

Example: PUT Object

The following example sends a PUT request to upload an object. The signature calculations assume the following:

  • You are uploading a 65 KB text file, and the file content is a one-character string made up of the letter 'a'.

  • The chunk size is 64 KB. As a result, the payload will be uploaded in three chunks, 64 KB, 1 KB, and the final chunk with 0 bytes of chunk data.

  • The resulting object has the key name chunkObject.txt.

  • You are requesting REDUCED_REDUNDANCY as the storage class by adding the x-amz-storage-class request header.

For information about the API action, see PUT Object. The general request syntax is as follows:

Copy
PUT /examplebucket/chunkObject.txt HTTP/1.1 Host: s3.amazonaws.com x-amz-date: 20130524T000000Z x-amz-storage-class: REDUCED_REDUNDANCY Authorization: SignatureToBeCalculated x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD Content-Encoding: aws-chunked x-amz-decoded-content-length: 66560 Content-Length: 66824 <Payload>

The following steps show signature calculations.

  1. Seed signature — Create String to Sign

    1. CanonicalRequest

      Copy
      PUT /examplebucket/chunkObject.txt content-encoding:aws-chunked content-length:66824 host:s3.amazonaws.com x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD x-amz-date:20130524T000000Z x-amz-decoded-content-length:66560 x-amz-storage-class:REDUCED_REDUNDANCY content-encoding;content-length;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-storage-class STREAMING-AWS4-HMAC-SHA256-PAYLOAD

      In the canonical request, the third line is empty because there are no query parameters in the request. The last line is the constant string provided as the value of the hashed Payload which should be same as the value of x-amz-content-sha256 header.

    2. StringToSign

      Copy
      AWS4-HMAC-SHA256 20130524T000000Z 20130524/us-east-1/s3/aws4_request cee3fed04b70f867d036f722359b0b1f2f0e5dc0efadbc082b76c4c60e316455

      Note

      For information about each of line in the string to sign, see the diagram that explains seed signature calculation.

  2. SigningKey

    Copy
    signing key = HMAC-SHA256(HMAC-SHA256(HMAC-SHA256(HMAC-SHA256("AWS4" + "<YourSecretAccessKey>","20130524"),"us-east-1"),"s3"),"aws4_request")
  3. Seed Signature

    Copy
    4f232c4386841ef735655705268965c44a0e4690baa4adea153f7db9fa80a0a9
  4. Authorization header

    The resulting Authorization header is as follows:

    Copy
    AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request,SignedHeaders=content-encoding;content-length;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-storage-class,Signature=4f232c4386841ef735655705268965c44a0e4690baa4adea153f7db9fa80a0a9
  5. Chunk 1: (65536 bytes, with value 97 for letter 'a')

    1. Chunk string to sign:

      Copy
      AWS4-HMAC-SHA256-PAYLOAD 20130524T000000Z 20130524/us-east-1/s3/aws4_request 4f232c4386841ef735655705268965c44a0e4690baa4adea153f7db9fa80a0a9 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 bf718b6f653bebc184e1479f1935b8da974d701b893afcf49e701f3e2f9f9c5a

      Note

      To information about each line in the string to sign, see the preceding diagram that show various components of the string to sign (for example, the last three lines are, previous-signature, hash(""), and hash(current-chunk-data)).

    2. Chunk signature:

      Copy
      ad80c730a21e5b8d04586a2213dd63b9a0e99e0e2307b0ade35a65485a288648
    3. Chunk data sent:

      Copy
      10000;chunk-signature=ad80c730a21e5b8d04586a2213dd63b9a0e99e0e2307b0ade35a65485a288648 <65536-bytes>
  6. Chunk 2: (1024 bytes, with value 97 for letter 'a')

    1. Chunk string to sign:

      Copy
      AWS4-HMAC-SHA256-PAYLOAD 20130524T000000Z 20130524/us-east-1/s3/aws4_request ad80c730a21e5b8d04586a2213dd63b9a0e99e0e2307b0ade35a65485a288648 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2edc986847e209b4016e141a6dc8716d3207350f416969382d431539bf292e4a
    2. Chunk signature:

      Copy
      0055627c9e194cb4542bae2aa5492e3c1575bbb81b612b7d234b86a503ef5497
    3. Chunk data sent:

      Copy
      400;chunk-signature=0055627c9e194cb4542bae2aa5492e3c1575bbb81b612b7d234b86a503ef5497 <1024 bytes>
  7. Chunk 3: (0 byte data)

    1. Chunk string to sign:

      Copy
      AWS4-HMAC-SHA256-PAYLOAD 20130524T000000Z 20130524/us-east-1/s3/aws4_request 0055627c9e194cb4542bae2aa5492e3c1575bbb81b612b7d234b86a503ef5497 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    2. Chunk signature:

      Copy
      b6c6ea8a5354eaf15b3cb7646744f4275b71ea724fed81ceb9323e279d449df9
    3. Chunk data sent:

      Copy
      0;chunk-signature=b6c6ea8a5354eaf15b3cb7646744f4275b71ea724fed81ceb9323e279d449df9