Using Amazon S3 multipart uploads with AWS SDK for PHP Version 3

With a single PutObject operation, you can upload objects up to 5 GB in size. However, by using the multipart upload methods (for example, CreateMultipartUpload, UploadPart, CompleteMultipartUpload, AbortMultipartUpload), you can upload objects from 5 MB to 5 TB in size.

The following example shows how to:

All the example code for the AWS SDK for PHP is available here on GitHub.

Credentials

Before running the example code, configure your AWS credentials, as described in Setting credentials. Then import the AWS SDK for PHP, as described in Basic usage.

Object uploader

If you’re not sure whether PutObject or MultipartUploader is best for the task, use ObjectUploader. ObjectUploader uploads a large file to Amazon S3 using either PutObject or MultipartUploader, depending on what is best based on the payload size.

require 'vendor/autoload.php';

use Aws\S3\S3Client;
use Aws\Exception\AwsException;
use Aws\S3\ObjectUploader;

Sample Code

$s3Client = new S3Client([
    'profile' => 'default',
    'region' => 'us-east-2',
    'version' => '2006-03-01'
]);

$bucket = 'your-bucket';
$key = 'my-file.zip';

// Using stream instead of file path
$source = fopen('/path/to/large/file.zip', 'rb');

$uploader = new ObjectUploader(
    $s3Client,
    $bucket,
    $key,
    $source
);

do {
    try {
        $result = $uploader->upload();
        if ($result["@metadata"]["statusCode"] == '200') {
            print('<p>File successfully uploaded to ' . $result["ObjectURL"] . '.</p>');
        }
        print($result);
    } catch (MultipartUploadException $e) {
        rewind($source);
        $uploader = new MultipartUploader($s3Client, $source, [
            'state' => $e->getState(),
        ]);
    }
} while (!isset($result));

fclose($source);

Configuration

The ObjectUploader object constructor accepts the following arguments:

MultipartUploader

Multipart uploads are designed to improve the upload experience for larger objects. They enable you to upload objects in parts independently, in any order, and in parallel.

Amazon S3 customers are encouraged to use multipart uploads for objects greater than 100 MB.

MultipartUploader object

The SDK has a special MultipartUploader object that simplifies the multipart upload process.

Imports

require 'vendor/autoload.php';

use Aws\S3\S3Client;
use Aws\Exception\AwsException;
use Aws\S3\MultipartUploader;
use Aws\Exception\MultipartUploadException;

Sample Code

$s3Client = new S3Client([
    'profile' => 'default',
    'region' => 'us-west-2',
    'version' => '2006-03-01'
]);

// Use multipart upload
$source = '/path/to/large/file.zip';
$uploader = new MultipartUploader($s3Client, $source, [
    'bucket' => 'your-bucket',
    'key' => 'my-file.zip',
]);

try {
    $result = $uploader->upload();
    echo "Upload complete: {$result['ObjectURL']}\n";
} catch (MultipartUploadException $e) {
    echo $e->getMessage() . "\n";
}
 

The uploader creates a generator of part data, based on the provided source and configuration, and attempts to upload all parts. If some part uploads fail, the uploader continues to upload later parts until the entire source data is read. Afterwards, the uploader retries to upload the failed parts or throws an exception containing information about the parts that failed to upload.

Customizing a multipart upload

You can set custom options on the CreateMultipartUpload, UploadPart, and CompleteMultipartUpload operations executed by the multipart uploader via callbacks passed to its constructor.

Imports

require 'vendor/autoload.php';

use Aws\S3\S3Client;
use Aws\Exception\AwsException;
use Aws\S3\MultipartUploader;
use Aws\Exception\MultipartUploadException;

Sample Code

// Create an S3Client
$s3Client = new S3Client([
    'profile' => 'default',
    'region' => 'us-west-2',
    'version' => '2006-03-01'
]);


// Customizing a multipart upload
$source = '/path/to/large/file.zip';
$uploader = new MultipartUploader($s3Client, $source, [
    'bucket' => 'your-bucket',
    'key' => 'my-file.zip',
    'before_initiate' => function (\Aws\Command $command) {
        // $command is a CreateMultipartUpload operation
        $command['CacheControl'] = 'max-age=3600';
    },
    'before_upload' => function (\Aws\Command $command) {
        // $command is an UploadPart operation
        $command['RequestPayer'] = 'requester';
    },
    'before_complete' => function (\Aws\Command $command) {
        // $command is a CompleteMultipartUpload operation
        $command['RequestPayer'] = 'requester';
    },
]);
 

Manual garbage collection between part uploads

If you are hitting the memory limit with large uploads, this may be due to cyclic references generated by the SDK not yet having been collected by the PHP garbage collector when your memory limit was hit. Manually invoking the collection algorithm between operations may allow the cycles to be collected before hitting that limit. The following example invokes the collection algorithm using a callback before each part upload. Note that invoking the garbage collector does come with a performance cost, and optimal usage will depend on your use case and environment.

$uploader = new MultipartUploader($client, $source, [
   'bucket' => 'your-bucket',
   'key' => 'your-key',
   'before_upload' => function(\Aws\Command $command) {
      gc_collect_cycles();
   }
]);

Recovering from errors

When an error occurs during the multipart upload process, a MultipartUploadException is thrown. This exception provides access to the UploadState object, which contains information about the multipart upload’s progress. The UploadState can be used to resume an upload that failed to complete.

Imports

require 'vendor/autoload.php';

use Aws\S3\S3Client;
use Aws\Exception\AwsException;
use Aws\S3\MultipartUploader;
use Aws\Exception\MultipartUploadException;

Sample Code

// Create an S3Client
$s3Client = new S3Client([
    'profile' => 'default',
    'region' => 'us-west-2',
    'version' => '2006-03-01'
]);

$source = '/path/to/large/file.zip';
$uploader = new MultipartUploader($s3Client, $source, [
    'bucket' => 'your-bucket',
    'key' => 'my-file.zip',
]);

//Recover from errors
do {
    try {
        $result = $uploader->upload();
    } catch (MultipartUploadException $e) {
        $uploader = new MultipartUploader($s3Client, $source, [
            'state' => $e->getState(),
        ]);
    }
} while (!isset($result));

//Abort a multipart upload if failed
try {
    $result = $uploader->upload();
} catch (MultipartUploadException $e) {
    // State contains the "Bucket", "Key", and "UploadId"
    $params = $e->getState()->getId();
    $result = $s3Client->abortMultipartUpload($params);
}

Resuming an upload from an UploadState attempts to upload parts that are not already uploaded. The state object tracks the missing parts, even if they are not consecutive. The uploader reads or seeks through the provided source file to the byte ranges that belong to the parts that still need to be uploaded.

UploadState objects are serializable, so you can also resume an upload in a different process. You can also get the UploadState object, even when you’re not handling an exception, by calling $uploader->getState().

Imports

require 'vendor/autoload.php';

use Aws\S3\S3Client;
use Aws\Exception\AwsException;
use Aws\S3\MultipartUploader;
use Aws\Exception\MultipartUploadException;

Sample Code

// Create an S3Client
$s3Client = new S3Client([
    'profile' => 'default',
    'region' => 'us-west-2',
    'version' => '2006-03-01'
]);

//Using stream instead of file path
$source = fopen('/path/to/large/file.zip', 'rb');
$uploader = new MultipartUploader($s3Client, $source, [
    'bucket' => 'your-bucket',
    'key' => 'my-file.zip',
]);

do {
    try {
        $result = $uploader->upload();
    } catch (MultipartUploadException $e) {
        rewind($source);
        $uploader = new MultipartUploader($s3Client, $source, [
            'state' => $e->getState(),
        ]);
    }
} while (!isset($result));
 

Aborting a multipart upload

A multipart upload can be aborted by retrieving the UploadId contained in the UploadState object and passing it to abortMultipartUpload.

try {
    $result = $uploader->upload();
} catch (MultipartUploadException $e) {
    // State contains the "Bucket", "Key", and "UploadId"
    $params = $e->getState()->getId();
    $result = $s3Client->abortMultipartUpload($params);
}

Asynchronous multipart uploads

Calling upload() on the MultipartUploader is a blocking request. If you are working in an asynchronous context, you can get a promise for the multipart upload.

require 'vendor/autoload.php';

use Aws\S3\S3Client;
use Aws\Exception\AwsException;
use Aws\S3\MultipartUploader;
use Aws\Exception\MultipartUploadException;

Sample Code

// Create an S3Client
$s3Client = new S3Client([
    'profile' => 'default',
    'region' => 'us-west-2',
    'version' => '2006-03-01'
]);


$source = '/path/to/large/file.zip';
$uploader = new MultipartUploader($s3Client, $source, [
    'bucket' => 'your-bucket',
    'key' => 'my-file.zip',
]);

$promise = $uploader->promise();
 

Configuration

The MultipartUploader object constructor accepts the following arguments:

Multipart copies

The AWS SDK for PHP also includes a MultipartCopy object that is used in a similar way to the MultipartUploader, but is designed for copying objects between 5 GB and 5 TB in size within Amazon S3.

require 'vendor/autoload.php';

use Aws\S3\S3Client;
use Aws\Exception\AwsException;
use Aws\S3\MultipartCopy;
use Aws\Exception\MultipartUploadException;

Sample Code

// Create an S3Client
$s3Client = new S3Client([
    'profile' => 'default',
    'region' => 'us-west-2',
    'version' => '2006-03-01'
]);


//Copy objects within S3
$copier = new MultipartCopy($s3Client, '/bucket/key?versionId=foo', [
    'bucket' => 'your-bucket',
    'key' => 'my-file.zip',
]);

try {
    $result = $copier->copy();
    echo "Copy complete: {$result['ObjectURL']}\n";
} catch (MultipartUploadException $e) {
    echo $e->getMessage() . "\n";
}