Menu
AWS SDK for Java
Developer Guide

Using TransferManager for Amazon S3 Operations

You can use the AWS SDK for JavaTransferManager class to reliably transfer files from the local environment to Amazon S3 and to copy objects from one S3 location to another. TransferManager can get the progress of a transfer and pause or resume uploads and downloads.

Note

These code snippets assume that you understand the material in Using the AWS SDK for Java and have configured default AWS credentials using the information in Set up AWS Credentials and Region for Development.

Upload Files and Directories

TransferManager can upload files, file lists, and directories to any Amazon S3 buckets that you've previously created.

Upload a Single File

Call the TransferManagerupload method, providing an Amazon S3 bucket name, a key (object) name, and a standard Java File object that represents the file to upload.

Imports

Copy
import com.amazonaws.AmazonServiceException; import com.amazonaws.services.s3.transfer.TransferManager; import com.amazonaws.services.s3.transfer.Upload; import java.io.File;

Code

Copy
File f = new File(file_path); TransferManager xfer_mgr = new TransferManager(); try { Upload xfer = xfer_mgr.upload(bucket_name, key_name, f); // loop with Transfer.isDone() // or block with Transfer.waitForCompletion() } catch (AmazonServiceException e) { System.err.println(e.getErrorMessage()); System.exit(1); } xfer_mgr.shutdownNow();

The upload method returns immediately, providing an Upload object to use to check the transfer state or to wait for it to complete.

See Wait for a Transfer to Complete for information about using waitForCompletion to successfully complete a transfer before calling TransferManager's shutdownNow method. While waiting for the transfer to complete, you can poll or listen for updates about its status and progress. See Get Transfer Status and Progress for more information.

See the complete example.

Upload a List of Files

To upload multiple files in one operation, call the TransferManageruploadFileList method, providing the following:

  • An Amazon S3 bucket name

  • A key prefix to prepend to the names of the created objects (the path within the bucket in which to place the objects)

  • A File object that represents the relative directory from which to create file paths

  • A List object containing a set of File objects to upload

Imports

Copy
import com.amazonaws.AmazonServiceException; import com.amazonaws.services.s3.transfer.TransferManager; import com.amazonaws.services.s3.transfer.MultipleFileUpload; import java.io.File; import java.util.ArrayList;

Code

Copy
ArrayList<File> files = new ArrayList<File>(); for (String path : file_paths) { files.add(new File(path)); } TransferManager xfer_mgr = new TransferManager(); try { MultipleFileUpload xfer = xfer_mgr.uploadFileList(bucket_name, key_prefix, new File("."), files); // loop with Transfer.isDone() // or block with Transfer.waitForCompletion() } catch (AmazonServiceException e) { System.err.println(e.getErrorMessage()); System.exit(1); } xfer_mgr.shutdownNow();

See Wait for a Transfer to Complete for information about using waitForCompletion to successfully complete a transfer before calling TransferManager's shutdownNow method. While waiting for the transfer to complete, you can poll or listen for updates about its status and progress. See Get Transfer Status and Progress for more information.

The MultipleFileUpload object returned by uploadFileList can be used to query the transfer state or progress. See Poll the Current Progress of a Transfer and Get Transfer Progress with a ProgressListener for more information.

You can also use MultipleFileUpload's getSubTransfers method to get the individual Upload objects for each file being transferred. For more information, see Get the Progress of Subtransfers.

See the complete example.

Upload a Directory

You can use TransferManager's uploadDirectory method to upload an entire directory of files, with the option to copy files in subdirectories recursively. You provide an Amazon S3 bucket name, an S3 key prefix, a File object representing the local directory to copy, and a boolean value indicating whether you want to copy subdirectories recursively (true or false).

Imports

Copy
import com.amazonaws.AmazonServiceException; import com.amazonaws.services.s3.transfer.TransferManager; import com.amazonaws.services.s3.transfer.Upload; import com.amazonaws.services.s3.transfer.MultipleFileUpload; import java.io.File;

Code

Copy
TransferManager xfer_mgr = new TransferManager(); try { MultipleFileUpload xfer = xfer_mgr.uploadDirectory(bucket_name, key_prefix, new File(dir_path), recursive); // loop with Transfer.isDone() // or block with Transfer.waitForCompletion() } catch (AmazonServiceException e) { System.err.println(e.getErrorMessage()); System.exit(1); } xfer_mgr.shutdownNow();

See Wait for a Transfer to Complete for information about using waitForCompletion to successfully complete a transfer before calling TransferManager's shutdownNow method. While waiting for the transfer to complete, you can poll or listen for updates about its status and progress. See Get Transfer Status and Progress for more information.

The MultipleFileUpload object returned by uploadFileList can be used to query the transfer state or progress. See Poll the Current Progress of a Transfer and Get Transfer Progress with a ProgressListener for more information.

You can also use MultipleFileUpload's getSubTransfers method to get the individual Upload objects for each file being transferred. For more information, see Get the Progress of Subtransfers.

See the complete example.

Download Files or Directories

Use the TransferManager class to download either a single file (Amazon S3 object) or a directory (an Amazon S3 bucket name followed by an object prefix) from Amazon S3.

Download a Single File

Use the TransferManager's download method, providing the Amazon S3 bucket name containing the object you want to download, the key (object) name, and a File object that represents the file to create on your local system.

Imports

Copy
import com.amazonaws.AmazonServiceException; import com.amazonaws.services.s3.transfer.TransferManager; import com.amazonaws.services.s3.transfer.Download; import java.io.File;

Code

Copy
File f = new File(file_path); TransferManager xfer_mgr = new TransferManager(); try { Download xfer = xfer_mgr.download(bucket_name, key_name, f); // loop with Transfer.isDone() // or block with Transfer.waitForCompletion() } catch (AmazonServiceException e) { System.err.println(e.getErrorMessage()); System.exit(1); } xfer_mgr.shutdownNow();

See Wait for a Transfer to Complete for information about using waitForCompletion to successfully complete a transfer before calling TransferManager's shutdownNow method. While waiting for the transfer to complete, you can poll or listen for updates about its status and progress. See Get Transfer Status and Progress for more information.

See the complete example.

Download a Directory

To download a set of files that share a common key prefix (analagous to a directory on a file system) from Amazon S3, use the TransferManagerdownloadDirectory method. The method takes the Amazon S3 bucket name containing the objects you want to download, the object prefix shared by all of the objects, and a File object that represents the directory to download the files into on your local system. If the named directory doesn't exist yet, it will be created.

Imports

Copy
import com.amazonaws.AmazonServiceException; import com.amazonaws.services.s3.transfer.TransferManager; import com.amazonaws.services.s3.transfer.MultipleFileDownload; import java.io.File;

Code

Copy
TransferManager xfer_mgr = new TransferManager(); try { MultipleFileDownload xfer = xfer_mgr.downloadDirectory( bucket_name, key_prefix, new File(dir_path)); // loop with Transfer.isDone() // or block with Transfer.waitForCompletion() } catch (AmazonServiceException e) { System.err.println(e.getErrorMessage()); System.exit(1); } xfer_mgr.shutdownNow();

See Wait for a Transfer to Complete for information about using waitForCompletion to successfully complete a transfer before calling TransferManager's shutdownNow method. While waiting for the transfer to complete, you can poll or listen for updates about its status and progress. See Get Transfer Status and Progress for more information.

See the complete example.

Copy Objects

To copy an object from one S3 bucket to another, use the TransferManagercopy method.

Imports

Copy
import com.amazonaws.AmazonServiceException; import com.amazonaws.services.s3.transfer.Copy; import com.amazonaws.services.s3.transfer.TransferManager;

Code

Copy
TransferManager xfer_mgr = new TransferManager(); try { Copy xfer = xfer_mgr.copy(from_bucket, from_key, to_bucket, to_key); // loop with Transfer.isDone() // or block with Transfer.waitForCompletion() } catch (AmazonServiceException e) { System.err.println(e.getErrorMessage()); System.exit(1); } xfer_mgr.shutdownNow();

See the complete example.

Wait for a Transfer to Complete

If your application (or thread) can block until the transfer completes, you can use the Transfer interface's waitForCompletion method to block until the transfer is complete or an exception occurs.

Copy
try { xfer.waitForCompletion(); } catch (AmazonServiceException e) { System.err.println("Amazon service error: " + e.getMessage()); System.exit(1); } catch (AmazonClientException e) { System.err.println("Amazon client error: " + e.getMessage()); System.exit(1); } catch (InterruptedException e) { System.err.println("Transfer interrupted: " + e.getMessage()); System.exit(1); }

You get progress of transfers if you poll for events before calling waitForCompletion, implement a polling mechanism on a separate thread, or receive progress updates asynchronously using a ProgressListener.

See the complete example.

Get Transfer Status and Progress

Each of the classes returned by the TransferManagerupload*, download*, and copy methods returns an instance of one of the following classes, depending on whether it's a single-file or multiple-file operation.

Class Returned by

Copy

copy

Download

download

MultipleFileDownload

downloadDirectory

Upload

upload

MultipleFileUpload

uploadFileList, uploadDirectory

All of these classes implement the Transfer interface. Transfer provides useful methods to get the progress of a transfer, pause or resume the transfer, and get the transfer's current or final status.

Poll the Current Progress of a Transfer

This loop prints the progress of a transfer, examines its current progress while running and, when complete, prints its final state.

Imports

Copy
import com.amazonaws.services.s3.transfer.TransferProgress;

Code

Copy
do { try { Thread.sleep(100); } catch (InterruptedException e) { return; } TransferProgress progress = xfer.getProgress(); long so_far = progress.getBytesTransferred(); long total = progress.getTotalBytesToTransfer(); double pct = progress.getPercentTransferred(); } while (xfer.isDone() == false);

See the complete example.

Get Transfer Progress with a ProgressListener

You can attach a ProgressListener to any transfer by using the Transfer interface's addProgressListener method.

A ProgressListener requires only one method, progressChanged, which takes a ProgressEvent object. You can use the object to get the total bytes of the operation by calling its getBytes method, and the number of bytes transferred so far by calling getBytesTransferred.

Imports

Copy
import com.amazonaws.AmazonServiceException; import com.amazonaws.event.ProgressEvent; import com.amazonaws.event.ProgressListener; import com.amazonaws.services.s3.transfer.Upload; import java.io.File;

Code

Copy
File f = new File(file_path); TransferManager xfer_mgr = new TransferManager(); try { Upload u = xfer_mgr.upload(bucket_name, key_name, f); u.addProgressListener(new ProgressListener() { public void progressChanged(ProgressEvent e) { double pct = e.getBytesTransferred() * 100.0 / e.getBytes(); } }); // block with Transfer.waitForCompletion() TransferState xfer_state = u.getState(); System.out.println(": " + xfer_state); } catch (AmazonServiceException e) { System.err.println(e.getErrorMessage()); System.exit(1); } xfer_mgr.shutdownNow();

See the complete example.

Get the Progress of Subtransfers

The MultipleFileUpload class can return information about its subtransfers by calling its getSubTransfers method. It returns an unmodifiable Collection of Upload objects that provide the individual transfer status and progress of each subtransfer.

Imports

Copy
import com.amazonaws.services.s3.transfer.Upload; import com.amazonaws.services.s3.transfer.MultipleFileUpload; import java.util.ArrayList; import java.util.Collection;

Code

Copy
Collection<? extends Upload> sub_xfers = new ArrayList<Upload>(); sub_xfers = multi_upload.getSubTransfers();

See the complete example.

More Info