Changes in Amazon S3 Transfer Manager from version 1 to version 2 - AWS SDK for Java 2.x
High-level changesConfiguration changesBehavior changes

Changes in Amazon S3 Transfer Manager from version 1 to version 2

This topic details the changes in the Amazon S3 Transfer Manager from version 1 (v1) to version 2 (v2).

High-level changes

Change v1 v2

Maven dependencies

 
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-bom</artifactId>
            <version>1.12.6911</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>
<dependencies>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-s3</artifactId>
    </dependency>
</dependencies>
 
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>bom</artifactId>
            <version>2.27.212</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>
<dependencies>
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>s3-transfer-manager</artifactId>
    </dependency>
// Add the following if using the 
// AWS CRT-based S3 client.
    <dependency>
        <groupId>software.amazon.awssdk.crt</groupId>
        <artifactId>aws-crt</artifactId>
        <version>0.29.143</version>
    </dependency>
</dependencies>
Package name com.amazonaws.services.s3.transfer software.amazon.awssdk.transfer.s3
Class name

TransferManager

S3TransferManager

1 Latest version. 2 Latest version. 3Latest version.

Configuration changes

The configuration changes that you need to set for the v2 transfer manager depend on which S3 client that you use. You have the choice of the AWS CRT-based S3 client or the standard Java-based S3 async client. For information about the differences, see the S3 clients in the AWS SDK for Java 2.x topic.

Use the AWS CRT-based S3 client
Setting v1 v2 - Transfer Manager using AWS CRT-based S3 client

(get a builder)

 
TransferManagerBuilder tmBuilder = 
   TransferManagerBuilder.standard();
 
S3TransferManager.Builder tmBuilder  = 
  S3TransferManager.builder();

S3 client

 
tmBuilder.withS3Client(...);
tmBuilder.setS3Client(...);
 
tmBuilder.s3Client(...);

Executor

 
tmBuilder.withExecutorFactory(...);
tmBuilder.setExecutorFactory(...);
 
tmBuilder.executor(...);

Shutdown thread pools

 
tmBuilder.withShutDownThreadPools(...);
tmBuilder.setShutdownThreadPools(...);
 Not supported. The provided executor will not be shut down when the S3TransferManager is closed

Minimum upload part size

 
tmBuilder.withMinimumUploadPartSize(...);
tmBuilder.setMinimumUploadPartSize(...);
 
S3AsyncClient s3 = S3AsyncClient.crtBuilder().
      minimumPartSizeInBytes(...).build();

tmBuilder.s3Client(s3);

Multipart upload threshold

 
tmBuilder.withMinimumUploadPartSize(...);
tmBuilder.setMinimumUploadPartSize(...);
 
S3AsyncClient s3 = S3AsyncClient.crtBuilder().
      thresholdInBytes(...).build();

tmBuilder.s3Client(s3);

Minimum copy part size

 
tmBuilder.withMinimumUploadPartSize(...);
tmBuilder.setMinimumUploadPartSize(...);
 
S3AsyncClient s3 = S3AsyncClient.crtBuilder().
      minimumPartSizeInBytes(...).build();

tmBuilder.s3Client(s3);

Multipart copy threshold

 
tmBuilder.withMinimumUploadPartSize(...);
tmBuilder.setMinimumUploadPartSize(...);
 
S3AsyncClient s3 = S3AsyncClient.crtBuilder().
      thresholdInBytes(...).build();

tmBuilder.s3Client(s3);

Disable parallel downloads

 
tmBuilder.withDisableParallelDownloads(...);
tmBuilder.setDisableParallelDownloads(...);
 Disable parallel downloads by passing a standard Java-based S3 client with multipart disabled (default) to the transfer manager.
S3AsyncClient s3 =
   S3AsyncClient.builder().build();

tmBuilder.s3Client(s3);

Always calculate multipart md5

 
tmBuilder.withAlwaysCalculateMultipartMd5(...);
tmBuilder.setAlwaysCalculateMultipartMd5(...);
 Not supported.
Use Java-based S3 async client
Setting v1 v2 - Transfer Manager using Java-based S3 async client

(get a builder)

 
TransferManagerBuilder tmBuilder = 
   TransferManagerBuilder.standard();
 
S3TransferManager.Builder tmBuilder  = 
  S3TransferManager.builder();

S3 client

 
tmBuilder.withS3Client(...);
tmBuilder.setS3Client(...);
 
tmBuilder.s3Client(...);

Executor

 
tmBuilder.withExecutorFactory(...);
tmBuilder.setExecutorFactory(...);
 
tmBuilder.executor(...);

Shutdown thread pools

 
tmBuilder.withShutDownThreadPools(...);
tmBuilder.setShutdownThreadPools(...);
 Not supported. The provided executor will not be shut down when the S3TransferManager is closed

Minimum upload part size

 
tmBuilder.withMinimumUploadPartSize(...);
tmBuilder.setMinimumUploadPartSize(...);
 
S3AsyncClient s3 = S3AsyncClient.builder()
    .multipartConfiguration(cfg ->
        cfg.minimumPartSizeInBytes(...)).build();

tmBuilder.s3Client(s3);

Multipart upload threshold

 
tmBuilder.withMinimumUploadPartSize(...);
tmBuilder.setMinimumUploadPartSize(...);
 
S3AsyncClient s3 = S3AsyncClient.builder()
    .multipartConfiguration(cfg ->
        cfg.thresholdInBytes(...)).build();

tmBuilder.s3Client(s3);

Minimum copy part size

 
tmBuilder.withMinimumUploadPartSize(...);
tmBuilder.setMinimumUploadPartSize(...);
 
S3AsyncClient s3 = S3AsyncClient.builder()
    .multipartConfiguration(cfg ->
        cfg.minimumPartSizeInBytes(...)).build();

tmBuilder.s3Client(s3);

Multipart copy threshold

 
tmBuilder.withMinimumUploadPartSize(...);
tmBuilder.setMinimumUploadPartSize(...);
 
S3AsyncClient s3 = S3AsyncClient.builder()
    .multipartConfiguration(cfg ->
        cfg.thresholdInBytes(...)).build();

tmBuilder.s3Client(s3);

Disable parallel downloads

 
tmBuilder.withDisableParallelDownloads(...);
tmBuilder.setDisableParallelDownloads(...);
 Disable parallel downloads by passing a standard Java-based S3 client with multipart disabled (default) to the transfer manager.
S3AsyncClient s3 =
   S3AsyncClient.builder().build();

tmBuilder.s3Client(s3);

Always calculate multipart md5

 
tmBuilder.withAlwaysCalculateMultipartMd5(...);
tmBuilder.setAlwaysCalculateMultipartMd5(...);
 Not supported.

Behavior changes

Parallel transfer requirements

In the SDK for Java 2.x, the automatic parallel transfer feature (multipart upload/download) is available through the AWS CRT-based S3 client and the Java-based S3 async client. To use the AWS CRT-based S3 client, you must explicitly add the AWS Common Runtime (CRT) library dependency for the maximized performance. To use the Java-based S3 async client with multipart enabled, you must use the SDK at version 2.25.X <TODO> or higher.

The AWS CRT-based S3 client alone—without using S3TransferManager—provides maximized performance of parallel transfers. S3TransferManager v2 provides additional APIs that make it easier to transfer files and directories.

Parallel download via byte-range fetches

When the automatic parallel transfer feature is enabled, the S3 Transfer Manager v2 uses byte-range fetches to retrieve specific portions of the object in parallel (multipart download). The way an object is downloaded with v2 does not depend on how the object was originally uploaded. All downloads can benefit from high throughput and concurrency.

In contrast, with S3 Transfer Manager v1, it does matter how the object was originally uploaded. The S3 Transfer Manager v1 retrieves the parts of the object the same way that the parts were uploaded. If an object was originally uploaded as a single object, the S3 Transfer Manager v1 is not able to accelerate the downloading process by using sub-requests.

Failure behavior

With S3 Transfer Manager v1, a directory transfer request fails if any sub-request fails. Unlike v1, the future returned from S3 Transfer Manager v2 completes successfully even if some sub-requests fail.

As a result, you should check for errors in the response by using the CompletedDirectoryDownload.failedTransfers() method or CompletedDirectoryUpload.failedTransfers() method even when the future completes successfully.