We announced the upcoming end-of-support for AWS SDK for Java (v1). We recommend that you migrate to AWS SDK for Java v2. For dates, additional details, and information on how to migrate, please refer to the linked announcement.

Using TransferManager for Amazon S3 Operations

You can use the AWS SDK for Java TransferManager class to reliably transfer files from the local environment to Amazon S3 and to copy objects from one S3 location to another. TransferManager can get the progress of a transfer and pause or resume uploads and downloads.

Upload Files and Directories

TransferManager can upload files, file lists, and directories to any Amazon S3 buckets that you’ve previously created.

Upload a Single File

Call TransferManager’s upload method, providing an Amazon S3 bucket name, a key (object) name, and a standard Java File object that represents the file to upload.

Imports

import com.amazonaws.AmazonServiceException;
import com.amazonaws.services.s3.transfer.MultipleFileUpload;
import com.amazonaws.services.s3.transfer.TransferManager;
import com.amazonaws.services.s3.transfer.TransferManagerBuilder;
import com.amazonaws.services.s3.transfer.Upload;

import java.io.File;
import java.util.ArrayList;
import java.util.Arrays;

Code

File f = new File(file_path);
TransferManager xfer_mgr = TransferManagerBuilder.standard().build();
try {
    Upload xfer = xfer_mgr.upload(bucket_name, key_name, f);
    // loop with Transfer.isDone()
    XferMgrProgress.showTransferProgress(xfer);
    //  or block with Transfer.waitForCompletion()
    XferMgrProgress.waitForCompletion(xfer);
} catch (AmazonServiceException e) {
    System.err.println(e.getErrorMessage());
    System.exit(1);
}
xfer_mgr.shutdownNow();

The upload method returns immediately, providing an Upload object to use to check the transfer state or to wait for it to complete.

See Wait for a Transfer to Complete for information about using waitForCompletion to successfully complete a transfer before calling TransferManager’s shutdownNow method. While waiting for the transfer to complete, you can poll or listen for updates about its status and progress. See Get Transfer Status and Progress for more information.

See the complete example on GitHub.

Upload a List of Files

To upload multiple files in one operation, call the TransferManageruploadFileList method, providing the following:

Imports

import com.amazonaws.AmazonServiceException;
import com.amazonaws.services.s3.transfer.MultipleFileUpload;
import com.amazonaws.services.s3.transfer.TransferManager;
import com.amazonaws.services.s3.transfer.TransferManagerBuilder;
import com.amazonaws.services.s3.transfer.Upload;

import java.io.File;
import java.util.ArrayList;
import java.util.Arrays;

Code

ArrayList<File> files = new ArrayList<File>();
for (String path : file_paths) {
    files.add(new File(path));
}

TransferManager xfer_mgr = TransferManagerBuilder.standard().build();
try {
    MultipleFileUpload xfer = xfer_mgr.uploadFileList(bucket_name,
            key_prefix, new File("."), files);
    // loop with Transfer.isDone()
    XferMgrProgress.showTransferProgress(xfer);
    // or block with Transfer.waitForCompletion()
    XferMgrProgress.waitForCompletion(xfer);
} catch (AmazonServiceException e) {
    System.err.println(e.getErrorMessage());
    System.exit(1);
}
xfer_mgr.shutdownNow();

See Wait for a Transfer to Complete for information about using waitForCompletion to successfully complete a transfer before calling TransferManager’s shutdownNow method. While waiting for the transfer to complete, you can poll or listen for updates about its status and progress. See Get Transfer Status and Progress for more information.

The MultipleFileUpload object returned by uploadFileList can be used to query the transfer state or progress. See Poll the Current Progress of a Transfer and Get Transfer Progress with a ProgressListener for more information.

You can also use MultipleFileUpload's getSubTransfers method to get the individual Upload objects for each file being transferred. For more information, see Get the Progress of Subtransfers.

See the complete example on GitHub.

Upload a Directory

You can use TransferManager’s uploadDirectory method to upload an entire directory of files, with the option to copy files in subdirectories recursively. You provide an Amazon S3 bucket name, an S3 key prefix, a File object representing the local directory to copy, and a boolean value indicating whether you want to copy subdirectories recursively (true or false).

Imports

import com.amazonaws.AmazonServiceException;
import com.amazonaws.services.s3.transfer.MultipleFileUpload;
import com.amazonaws.services.s3.transfer.TransferManager;
import com.amazonaws.services.s3.transfer.TransferManagerBuilder;
import com.amazonaws.services.s3.transfer.Upload;

import java.io.File;
import java.util.ArrayList;
import java.util.Arrays;

Code

TransferManager xfer_mgr = TransferManagerBuilder.standard().build();
try {
    MultipleFileUpload xfer = xfer_mgr.uploadDirectory(bucket_name,
            key_prefix, new File(dir_path), recursive);
    // loop with Transfer.isDone()
    XferMgrProgress.showTransferProgress(xfer);
    // or block with Transfer.waitForCompletion()
    XferMgrProgress.waitForCompletion(xfer);
} catch (AmazonServiceException e) {
    System.err.println(e.getErrorMessage());
    System.exit(1);
}
xfer_mgr.shutdownNow();

See Wait for a Transfer to Complete for information about using waitForCompletion to successfully complete a transfer before calling TransferManager’s shutdownNow method. While waiting for the transfer to complete, you can poll or listen for updates about its status and progress. See Get Transfer Status and Progress for more information.

The MultipleFileUpload object returned by uploadFileList can be used to query the transfer state or progress. See Poll the Current Progress of a Transfer and Get Transfer Progress with a ProgressListener for more information.

You can also use MultipleFileUpload's getSubTransfers method to get the individual Upload objects for each file being transferred. For more information, see Get the Progress of Subtransfers.

See the complete example on GitHub.

Download Files or Directories

Use the TransferManager class to download either a single file (Amazon S3 object) or a directory (an Amazon S3 bucket name followed by an object prefix) from Amazon S3.

Download a Single File

Use the TransferManager’s download method, providing the Amazon S3 bucket name containing the object you want to download, the key (object) name, and a File object that represents the file to create on your local system.

Imports

import com.amazonaws.AmazonServiceException;
import com.amazonaws.services.s3.transfer.Download;
import com.amazonaws.services.s3.transfer.MultipleFileDownload;
import com.amazonaws.services.s3.transfer.TransferManager;
import com.amazonaws.services.s3.transfer.TransferManagerBuilder;

import java.io.File;

Code

File f = new File(file_path);
TransferManager xfer_mgr = TransferManagerBuilder.standard().build();
try {
    Download xfer = xfer_mgr.download(bucket_name, key_name, f);
    // loop with Transfer.isDone()
    XferMgrProgress.showTransferProgress(xfer);
    // or block with Transfer.waitForCompletion()
    XferMgrProgress.waitForCompletion(xfer);
} catch (AmazonServiceException e) {
    System.err.println(e.getErrorMessage());
    System.exit(1);
}
xfer_mgr.shutdownNow();

See Wait for a Transfer to Complete for information about using waitForCompletion to successfully complete a transfer before calling TransferManager’s shutdownNow method. While waiting for the transfer to complete, you can poll or listen for updates about its status and progress. See Get Transfer Status and Progress for more information.

See the complete example on GitHub.

Download a Directory

To download a set of files that share a common key prefix (analogous to a directory on a file system) from Amazon S3, use the TransferManagerdownloadDirectory method. The method takes the Amazon S3 bucket name containing the objects you want to download, the object prefix shared by all of the objects, and a File object that represents the directory to download the files into on your local system. If the named directory doesn’t exist yet, it will be created.

Imports

import com.amazonaws.AmazonServiceException;
import com.amazonaws.services.s3.transfer.Download;
import com.amazonaws.services.s3.transfer.MultipleFileDownload;
import com.amazonaws.services.s3.transfer.TransferManager;
import com.amazonaws.services.s3.transfer.TransferManagerBuilder;

import java.io.File;

Code

TransferManager xfer_mgr = TransferManagerBuilder.standard().build();

try {
    MultipleFileDownload xfer = xfer_mgr.downloadDirectory(
            bucket_name, key_prefix, new File(dir_path));
    // loop with Transfer.isDone()
    XferMgrProgress.showTransferProgress(xfer);
    // or block with Transfer.waitForCompletion()
    XferMgrProgress.waitForCompletion(xfer);
} catch (AmazonServiceException e) {
    System.err.println(e.getErrorMessage());
    System.exit(1);
}
xfer_mgr.shutdownNow();

See Wait for a Transfer to Complete for information about using waitForCompletion to successfully complete a transfer before calling TransferManager’s shutdownNow method. While waiting for the transfer to complete, you can poll or listen for updates about its status and progress. See Get Transfer Status and Progress for more information.

See the complete example on GitHub.

Copy Objects

To copy an object from one S3 bucket to another, use the TransferManagercopy method.

Imports

import com.amazonaws.AmazonServiceException;
import com.amazonaws.services.s3.transfer.Copy;
import com.amazonaws.services.s3.transfer.TransferManager;
import com.amazonaws.services.s3.transfer.TransferManagerBuilder;

Code

System.out.println("Copying s3 object: " + from_key);
System.out.println("      from bucket: " + from_bucket);
System.out.println("     to s3 object: " + to_key);
System.out.println("        in bucket: " + to_bucket);

TransferManager xfer_mgr = TransferManagerBuilder.standard().build();
try {
    Copy xfer = xfer_mgr.copy(from_bucket, from_key, to_bucket, to_key);
    // loop with Transfer.isDone()
    XferMgrProgress.showTransferProgress(xfer);
    // or block with Transfer.waitForCompletion()
    XferMgrProgress.waitForCompletion(xfer);
} catch (AmazonServiceException e) {
    System.err.println(e.getErrorMessage());
    System.exit(1);
}
xfer_mgr.shutdownNow();

See the complete example on GitHub.

Wait for a Transfer to Complete

If your application (or thread) can block until the transfer completes, you can use the Transfer interface’s waitForCompletion method to block until the transfer is complete or an exception occurs.

try {
    xfer.waitForCompletion();
} catch (AmazonServiceException e) {
    System.err.println("Amazon service error: " + e.getMessage());
    System.exit(1);
} catch (AmazonClientException e) {
    System.err.println("Amazon client error: " + e.getMessage());
    System.exit(1);
} catch (InterruptedException e) {
    System.err.println("Transfer interrupted: " + e.getMessage());
    System.exit(1);
}

You get progress of transfers if you poll for events before calling waitForCompletion, implement a polling mechanism on a separate thread, or receive progress updates asynchronously using a ProgressListener.

See the complete example on GitHub.

Get Transfer Status and Progress

Each of the classes returned by the TransferManagerupload*, download*, and copy methods returns an instance of one of the following classes, depending on whether it’s a single-file or multiple-file operation.

All of these classes implement the Transfer interface. Transfer provides useful methods to get the progress of a transfer, pause or resume the transfer, and get the transfer’s current or final status.

Poll the Current Progress of a Transfer

This loop prints the progress of a transfer, examines its current progress while running and, when complete, prints its final state.

Imports

import com.amazonaws.AmazonClientException;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.event.ProgressEvent;
import com.amazonaws.event.ProgressListener;
import com.amazonaws.services.s3.transfer.*;
import com.amazonaws.services.s3.transfer.Transfer.TransferState;

import java.io.File;
import java.util.ArrayList;
import java.util.Collection;

Code

// print the transfer's human-readable description
System.out.println(xfer.getDescription());
// print an empty progress bar...
printProgressBar(0.0);
// update the progress bar while the xfer is ongoing.
do {
    try {
        Thread.sleep(100);
    } catch (InterruptedException e) {
        return;
    }
    // Note: so_far and total aren't used, they're just for
    // documentation purposes.
    TransferProgress progress = xfer.getProgress();
    long so_far = progress.getBytesTransferred();
    long total = progress.getTotalBytesToTransfer();
    double pct = progress.getPercentTransferred();
    eraseProgressBar();
    printProgressBar(pct);
} while (xfer.isDone() == false);
// print the final state of the transfer.
TransferState xfer_state = xfer.getState();
System.out.println(": " + xfer_state);

See the complete example on GitHub.

Get Transfer Progress with a ProgressListener

You can attach a ProgressListener to any transfer by using the Transfer interface’s addProgressListener method.

A ProgressListener requires only one method, progressChanged, which takes a ProgressEvent object. You can use the object to get the total bytes of the operation by calling its getBytes method, and the number of bytes transferred so far by calling getBytesTransferred.

Imports

import com.amazonaws.AmazonClientException;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.event.ProgressEvent;
import com.amazonaws.event.ProgressListener;
import com.amazonaws.services.s3.transfer.*;
import com.amazonaws.services.s3.transfer.Transfer.TransferState;

import java.io.File;
import java.util.ArrayList;
import java.util.Collection;

Code

File f = new File(file_path);
TransferManager xfer_mgr = TransferManagerBuilder.standard().build();
try {
    Upload u = xfer_mgr.upload(bucket_name, key_name, f);
    // print an empty progress bar...
    printProgressBar(0.0);
    u.addProgressListener(new ProgressListener() {
        public void progressChanged(ProgressEvent e) {
            double pct = e.getBytesTransferred() * 100.0 / e.getBytes();
            eraseProgressBar();
            printProgressBar(pct);
        }
    });
    // block with Transfer.waitForCompletion()
    XferMgrProgress.waitForCompletion(u);
    // print the final state of the transfer.
    TransferState xfer_state = u.getState();
    System.out.println(": " + xfer_state);
} catch (AmazonServiceException e) {
    System.err.println(e.getErrorMessage());
    System.exit(1);
}
xfer_mgr.shutdownNow();

See the complete example on GitHub.

Get the Progress of Subtransfers

The MultipleFileUpload class can return information about its subtransfers by calling its getSubTransfers method. It returns an unmodifiable Collection of Upload objects that provide the individual transfer status and progress of each sub-transfer.

Imports

import com.amazonaws.AmazonClientException;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.event.ProgressEvent;
import com.amazonaws.event.ProgressListener;
import com.amazonaws.services.s3.transfer.*;
import com.amazonaws.services.s3.transfer.Transfer.TransferState;

import java.io.File;
import java.util.ArrayList;
import java.util.Collection;

Code

Collection<? extends Upload> sub_xfers = new ArrayList<Upload>();
sub_xfers = multi_upload.getSubTransfers();

do {
    System.out.println("\nSubtransfer progress:\n");
    for (Upload u : sub_xfers) {
        System.out.println("  " + u.getDescription());
        if (u.isDone()) {
            TransferState xfer_state = u.getState();
            System.out.println("  " + xfer_state);
        } else {
            TransferProgress progress = u.getProgress();
            double pct = progress.getPercentTransferred();
            printProgressBar(pct);
            System.out.println();
        }
    }

    // wait a bit before the next update.
    try {
        Thread.sleep(200);
    } catch (InterruptedException e) {
        return;
    }
} while (multi_upload.isDone() == false);
// print the final state of the transfer.
TransferState xfer_state = multi_upload.getState();
System.out.println("\nMultipleFileUpload " + xfer_state);

See the complete example on GitHub.

More Info