Transfer files and directories with the Amazon S3 Transfer Manager
The Amazon S3 Transfer Manager is an open source, high level file transfer utility for the AWS SDK for Java 2.x. Use it to transfer files and directories to and from Amazon Simple Storage Service (Amazon S3).
When built on top of the AWS CRT-based S3 client, the S3 Transfer Manager can take advantage of performance improvements such as the multipart upload API and byte-range fetches.
With the S3 Transfer Manager, you can also monitor a transfer's progress in real time and pause the transfer for later execution.
Get started
Add dependencies to your build file
To use the S3 Transfer Manager with enhanced performance based on the AWS CRT-based S3 client, configure your build file with the following dependencies.
-
Use version
2.19.1
or higher of the SDK for Java 2.x. -
Add the
s3-transfer-manager
artifact as a dependency. -
Add the
aws-crt
artifact as a dependency at version0.20.3
or higher.
The following code example shows how to configure your project dependencies for Maven.
<project> <properties> <aws.sdk.version>
2.19.1
</aws.sdk.version> </properties> <dependencyManagement> <dependencies> <dependency> <groupId>software.amazon.awssdk</groupId> <artifactId>bom</artifactId> <version>${aws.sdk.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> <dependencies> <dependency> <groupId>software.amazon.awssdk</groupId> <artifactId>s3-transfer-manager</artifactId> </dependency> <dependency> <groupId>software.amazon.awssdk.crt</groupId> <artifactId>aws-crt</artifactId> <version>0.20.3
</version> </dependency> </dependencies> </project>
Search the Maven central repository for the most recent versions of the s3-transfer-manager
Create an instance of the S3 Transfer Manager
The following snippet shows how to create a S3TransferManager
S3TransferManager transferManager = S3TransferManager.create();
The following example shows how to configure a S3 Transfer Manager with custom settings. In this example, a AWS CRT-based S3AsyncClient instance is used as the underlying client for the S3 Transfer Manager.
S3AsyncClient s3AsyncClient = S3AsyncClient.crtBuilder() .credentialsProvider(DefaultCredentialsProvider.create()) .region(Region.US_EAST_1) .targetThroughputInGbps(20.0) .minimumPartSizeInBytes(8 * MB) .build(); S3TransferManager transferManager = S3TransferManager.builder() .s3Client(s3AsyncClient) .build();
Note
If the aws-crt
dependency is not included in the build file, the
S3 Transfer Manager is built on top of the standard S3 asynchronous client used in the
SDK for Java 2.x.
Upload a file to an S3 bucket
The following example shows a file upload example along with the optional use of a
LoggingTransferListener
To upload a file to Amazon S3 using the S3 Transfer Manager, pass an UploadFileRequest
S3TransferManager
's uploadFile
The FileUploaduploadFile
method represents
the upload process. After the request finishes, the CompletedFileUpload
public String uploadFile(S3TransferManager transferManager, String bucketName, String key, URI filePathURI) { UploadFileRequest uploadFileRequest = UploadFileRequest.builder() .putObjectRequest(b -> b.bucket(bucketName).key(key)) .source(Paths.get(filePathURI)) .build(); FileUpload fileUpload = transferManager.uploadFile(uploadFileRequest); CompletedFileUpload uploadResult = fileUpload.completionFuture().join(); return uploadResult.response().eTag(); }
import org.slf4j.Logger; import org.slf4j.LoggerFactory; import software.amazon.awssdk.transfer.s3.S3TransferManager; import software.amazon.awssdk.transfer.s3.model.CompletedFileUpload; import software.amazon.awssdk.transfer.s3.model.FileUpload; import software.amazon.awssdk.transfer.s3.model.UploadFileRequest; import software.amazon.awssdk.transfer.s3.progress.LoggingTransferListener; import java.net.URI; import java.net.URISyntaxException; import java.net.URL; import java.nio.file.Paths; import java.util.UUID;
Download a file from an S3 bucket
The following example shows a download example along with the optional use of a LoggingTransferListener
To download an object from an S3 bucket using the S3 Transfer Manager, build a DownloadFileRequest
The FileDownloadS3TransferManager
's
downloadFile
method represents the file transfer. After the download
completes, the CompletedFileDownload
public Long downloadFile(S3TransferManager transferManager, String bucketName, String key, String downloadedFileWithPath) { DownloadFileRequest downloadFileRequest = DownloadFileRequest.builder() .getObjectRequest(b -> b.bucket(bucketName).key(key)) .destination(Paths.get(downloadedFileWithPath)) .build(); FileDownload downloadFile = transferManager.downloadFile(downloadFileRequest); CompletedFileDownload downloadResult = downloadFile.completionFuture().join(); logger.info("Content length [{}]", downloadResult.response().contentLength()); return downloadResult.response().contentLength(); }
import org.slf4j.Logger; import org.slf4j.LoggerFactory; import software.amazon.awssdk.core.sync.RequestBody; import software.amazon.awssdk.transfer.s3.S3TransferManager; import software.amazon.awssdk.transfer.s3.model.CompletedFileDownload; import software.amazon.awssdk.transfer.s3.model.DownloadFileRequest; import software.amazon.awssdk.transfer.s3.model.FileDownload; import software.amazon.awssdk.transfer.s3.progress.LoggingTransferListener; import java.io.IOException; import java.net.URISyntaxException; import java.net.URL; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; import java.util.UUID;
Copy an Amazon S3 object to another bucket
The following example shows how to copy an object with the S3 Transfer Manager.
To begin the copy of an object from an S3 bucket to another bucket, create a basic
CopyObjectRequest
Next, wrap the basic CopyObjectRequest
in a CopyRequest
The Copy
object returned by the S3TransferManager
's
copy
method represents the copy process. After the copy process completes,
the CompletedCopy
public String copyObject(S3TransferManager transferManager, String bucketName, String key, String destinationBucket, String destinationKey) { CopyObjectRequest copyObjectRequest = CopyObjectRequest.builder() .sourceBucket(bucketName) .sourceKey(key) .destinationBucket(destinationBucket) .destinationKey(destinationKey) .build(); CopyRequest copyRequest = CopyRequest.builder() .copyObjectRequest(copyObjectRequest) .build(); Copy copy = transferManager.copy(copyRequest); CompletedCopy completedCopy = copy.completionFuture().join(); return completedCopy.response().copyObjectResult().eTag(); }
Note
To perform a cross-Region copy with the S3 Transfer Manager, enable
crossRegionAccessEnabled
on the AWS CRT-based S3 client builder as
shown in the following snippet.
S3AsyncClient s3AsyncClient = S3AsyncClient.crtBuilder() .crossRegionAccessEnabled(true) .build(); S3TransferManager transferManager = S3TransferManager.builder() .s3Client(s3AsyncClient) .build();
import org.slf4j.Logger; import org.slf4j.LoggerFactory; import software.amazon.awssdk.core.sync.RequestBody; import software.amazon.awssdk.services.s3.model.CopyObjectRequest; import software.amazon.awssdk.transfer.s3.S3TransferManager; import software.amazon.awssdk.transfer.s3.model.CompletedCopy; import software.amazon.awssdk.transfer.s3.model.Copy; import software.amazon.awssdk.transfer.s3.model.CopyRequest; import java.util.UUID;
Upload a local directory to an S3 bucket
The following example demonstrates how you can upload a local directory to S3.
Start by calling the uploadDirectoryS3TransferManager
instance, passing
in an UploadDirectoryRequest
The DirectoryUploadCompleteDirectoryUpload
object contains information about the results of
the transfer, including which files failed to transfer.
public Integer uploadDirectory(S3TransferManager transferManager, URI sourceDirectory, String bucketName) { DirectoryUpload directoryUpload = transferManager.uploadDirectory(UploadDirectoryRequest.builder() .source(Paths.get(sourceDirectory)) .bucket(bucketName) .build()); CompletedDirectoryUpload completedDirectoryUpload = directoryUpload.completionFuture().join(); completedDirectoryUpload.failedTransfers() .forEach(fail -> logger.warn("Object [{}] failed to transfer", fail.toString())); return completedDirectoryUpload.failedTransfers().size(); }
import org.slf4j.Logger; import org.slf4j.LoggerFactory; import software.amazon.awssdk.services.s3.model.ObjectIdentifier; import software.amazon.awssdk.transfer.s3.S3TransferManager; import software.amazon.awssdk.transfer.s3.model.CompletedDirectoryUpload; import software.amazon.awssdk.transfer.s3.model.DirectoryUpload; import software.amazon.awssdk.transfer.s3.model.UploadDirectoryRequest; import java.net.URI; import java.net.URISyntaxException; import java.net.URL; import java.nio.file.Paths; import java.util.UUID;
Download S3 bucket objects to a local directory
You can download the objects in an S3 bucket to a local directory as shown in the following example.
To download the objects in an S3 bucket to a local directory, begin by calling the
downloadDirectory
The DirectoryDownloadCompleteDirectoryDownload
object contains information about the results of
the transfer, including which files failed to transfer.
public Integer downloadObjectsToDirectory(S3TransferManager transferManager, URI destinationPathURI, String bucketName) { DirectoryDownload directoryDownload = transferManager.downloadDirectory(DownloadDirectoryRequest.builder() .destination(Paths.get(destinationPathURI)) .bucket(bucketName) .build()); CompletedDirectoryDownload completedDirectoryDownload = directoryDownload.completionFuture().join(); completedDirectoryDownload.failedTransfers() .forEach(fail -> logger.warn("Object [{}] failed to transfer", fail.toString())); return completedDirectoryDownload.failedTransfers().size(); }
import org.slf4j.Logger; import org.slf4j.LoggerFactory; import software.amazon.awssdk.core.sync.RequestBody; import software.amazon.awssdk.services.s3.model.ObjectIdentifier; import software.amazon.awssdk.transfer.s3.S3TransferManager; import software.amazon.awssdk.transfer.s3.model.CompletedDirectoryDownload; import software.amazon.awssdk.transfer.s3.model.DirectoryDownload; import software.amazon.awssdk.transfer.s3.model.DownloadDirectoryRequest; import java.io.IOException; import java.net.URI; import java.net.URISyntaxException; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; import java.util.HashSet; import java.util.Set; import java.util.UUID; import java.util.stream.Collectors;
See complete examples
GitHub contains the complete