AWS SDK for Java
Developer Guide

Retrieving Paginated Results

Many AWS operations return paginated results when the response object is too large to return in a single response. In the AWS SDK for Java 1.0, the response contained a token you had to use to retrieve the next page of results. New in the AWS SDK for Java 2.0 are autopagination methods that make multiple service calls to get the next page of results for you automatically. You only have to write code that processes the results. Additionally both types of methods have synchronous and asynchronous versions. See Asynchronous Programming for more detail about asynchronous clients.

The following examples use Amazon S3 and Amazon DynamoDB operations to demonstrate the various methods of retrieving your data from paginated responses.

Note

These code snippets assume that you understand the material in Using the AWS SDK for Java 2.0, and have configured default AWS credentials using the information in Set Up AWS Credentials and Region for Development.

Synchronous Pagination

These examples use the synchronous pagination methods for listing objects in an Amazon S3 bucket.

Iterate over Pages

Build a ListObjectsV2Request and provide a bucket name. Optionally you can provide the maximum number of keys to retrieve at one time. Pass it to the S3Client's listObjectsV2Paginator method. This method returns a ListObjectsV2Iterable object, which is an Iterable of the ListObjectsV2Response class.

The first example demonstrates using the paginator object to iterate through all the response pages with the stream method. You can directly stream over the response pages, convert the response stream to a stream of S3Object content, and then process the content of the Amazon S3 object.

Imports

import software.amazon.awssdk.regions.Region; import software.amazon.awssdk.services.s3.S3Client; import software.amazon.awssdk.services.s3.model.ListObjectsV2Request; import software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable;

Code

// Build the list objects request ListObjectsV2Request listReq = ListObjectsV2Request.builder() .bucket(bucket) .maxKeys(1) .build(); ListObjectsV2Iterable listRes = s3.listObjectsV2Paginator(listReq); // Process response pages listRes.stream() .flatMap(r -> r.contents().stream()) .forEach(content -> System.out.println(" Key: " + content.key() + " size = " + content.size()));

See the complete example on GitHub.

Iterate over Objects

The following examples show ways to iterate over the objects returned in the response instead of the pages of the response.

Use a Stream

Use the stream method on the response content to iterate over the paginated item collection.

Code

// Helper method to work with paginated collection of items directly listRes.contents().stream() .forEach(content -> System.out.println(" Key: " + content.key() + " size = " + content.size()));

See the complete example on GitHub.

Use a For Loop

Use a standard for loop to iterate through the contents of the response.

Code

// Use simple for loop if stream is not necessary for (S3Object content : listRes.contents()) { System.out.println(" Key: " + content.key() + " size = " + content.size()); }

See the complete example on GitHub.

Manual Pagination

If your use case requires it, manual pagination is still available. Use the next token in the response object for the subsequent requests. Here's an example using a while loop.

Code

// Use manual pagination ListObjectsV2Request listObjectsReqManual = ListObjectsV2Request.builder() .bucket(bucket) .maxKeys(1) .build(); boolean done = false; while (!done) { ListObjectsV2Response listObjResponse = s3.listObjectsV2(listObjectsReqManual); for (S3Object content : listObjResponse.contents()) { System.out.println(content.key()); } if (listObjResponse.nextContinuationToken() == null) { done = true; } listObjectsReqManual = listObjectsReqManual.toBuilder() .continuationToken(listObjResponse.nextContinuationToken()) .build(); }

See the complete example on GitHub.

Asynchronous Pagination

These examples use the asynchronous pagination methods for listing tables in DynamoDB. A manual pagination example is available in the Asynchronous Programming topic.

Iterate over Pages of Table Names

First, create an asynchronous DynamoDB client. Then, call the listTablesPaginator method to get a ListTablesPublisher. This is an implementation of the reactive streams Publisher interface. To learn more about the reactive streams model, see the Reactive Streams Github repo.

Call the subscribe method on the ListTablesPublisher and pass a subscriber implementation. In this example, the subscriber has an onNext method that requests one item at a time from the publisher. This is the method that is called repeatedly until all pages are retrieved. The onSubscribe method calls the Subscription.request method to initiate requests for data from the publisher. This method must be called to start getting data from the publisher. The onError method is triggered if an error occurs while retrieving data. Finally, the onComplete method is called when all pages have been requested.

Use a Subscriber

Imports

import java.util.List; import java.util.concurrent.CompletableFuture; import java.util.concurrent.ExecutionException; import org.reactivestreams.Subscriber; import org.reactivestreams.Subscription; import software.amazon.awssdk.core.async.SdkPublisher; import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient; import software.amazon.awssdk.services.dynamodb.model.ListTablesRequest; import software.amazon.awssdk.services.dynamodb.model.ListTablesResponse; import software.amazon.awssdk.services.dynamodb.paginators.ListTablesPublisher;

Code

final DynamoDbAsyncClient asyncClient = DynamoDbAsyncClient.create(); ListTablesRequest listTablesRequest = ListTablesRequest.builder().limit(3).build(); ListTablesPublisher publisher = asyncClient.listTablesPaginator(listTablesRequest); // A Subscription represents a one-to-one life-cycle of a Subscriber subscribing to a Publisher. publisher.subscribe(new Subscriber<ListTablesResponse>() { // Maintain a reference to the subscription object, which is required to request data from the publisher private Subscription subscription; @Override public void onSubscribe(Subscription s) { subscription = s; // Request method should be called to demand data. Here we request a single page subscription.request(1); } @Override public void onNext(ListTablesResponse response) { response.tableNames().forEach(System.out::println); // Once you process the current page, call the request method to signal that you are ready for next page subscription.request(1); } @Override public void onError(Throwable t) { // Called when an error has occurred while processing the requests } @Override public void onComplete() { // This indicates all the results are delivered and there are no more pages left } });

See the complete example on GitHub.

Use a For Loop

Use a for loop to iterate through the pages for simple use cases when creating a new subscriber might be too much overhead. The response publisher object has a forEach helper method for this purpose.

Code

// Use a for-loop for simple use cases CompletableFuture<Void> future = publisher.subscribe(response -> response.tableNames() .forEach(System.out::println)); future.get();

See the complete example on GitHub.

Iterate over Table Names

The following examples show ways to iterate over the objects returned in the response instead of the pages of the response. Similar to the synchronous result, the asynchronous result class has a method to interact with the underlying item collection. The return type of the convenience method is a publisher that can be used to request items across all pages.

Use a Subscriber

Code

// Creates a default client with credentials and regions loaded from the environment final DynamoDbAsyncClient asyncClient = DynamoDbAsyncClient.create(); ListTablesRequest listTablesRequest = ListTablesRequest.builder().limit(3).build(); ListTablesPublisher listTablesPublisher = asyncClient.listTablesPaginator(listTablesRequest); SdkPublisher<String> publisher = listTablesPublisher.tableNames(); // Use subscriber publisher.subscribe(new Subscriber<String>() { private Subscription subscription; @Override public void onSubscribe(Subscription s) { subscription = s; subscription.request(1); } @Override public void onNext(String tableName) { System.out.println(tableName); subscription.request(1); } @Override public void onError(Throwable t) { } @Override public void onComplete() { } });

See the complete example on GitHub.

Use a For Loop

Use the forEach convenience method to iterate through the results.

Code

// Use forEach CompletableFuture<Void> future = publisher.subscribe(System.out::println); future.get();

See the complete example on GitHub.

Use Third-party Library

You can use other third party libraries instead of implementing a custom subscriber. This example demonstrates using the RxJava implementation but any library that implements the reactive stream interfaces can be used. See the RxJava wiki page on Github for more information on that library.

To use the library, add it as a dependency. If using Maven, the example shows the POM snippet to use.

POM Entry

<dependency> <groupId>io.reactivex.rxjava2</groupId> <artifactId>rxjava</artifactId> <version>2.1.9</version> </dependency>

Import

import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient; import software.amazon.awssdk.services.dynamodb.model.ListTablesRequest; import software.amazon.awssdk.services.dynamodb.model.ListTablesResponse; import software.amazon.awssdk.services.dynamodb.paginators.ListTablesPublisher; import io.reactivex.Flowable;

Code

DynamoDbAsyncClient asyncClient = DynamoDbAsyncClient.create(); ListTablesPublisher publisher = asyncClient.listTablesPaginator(ListTablesRequest.builder() .build()); // The Flowable class has many helper methods that work with any reactive streams compatible publisher implementation List<String> tables = Flowable.fromPublisher(publisher) .flatMapIterable(ListTablesResponse::tableNames) .toList() .blockingGet(); System.out.println(tables);