Menu
Amazon Simple Queue Service
Developer Guide

Managing Large Amazon SQS Messages Using Amazon S3

You can manage Amazon SQS messages with Amazon S3. This is especially useful for storing and consuming messages with a message size of up to 2 GB. To manage Amazon SQS messages with Amazon S3, use the Amazon SQS Extended Client Library for Java. Specifically, you use this library to:

  • Specify whether messages are always stored in Amazon S3 or only when the size of a message exceeds 256 KB.

  • Send a message that references a single message object stored in an Amazon S3 bucket.

  • Get the corresponding message object from an Amazon S3 bucket.

  • Delete the corresponding message object from an Amazon S3 bucket.

Note

You can use the Amazon SQS Extended Client Library for Java to manage Amazon SQS messages using Amazon S3. However, you can't do this using the AWS CLI, the Amazon SQS console, the Amazon SQS HTTP API, or any of the AWS SDKs—except for the SDK for Java.

Prerequisites

To manage Amazon SQS messages with Amazon S3, you need the following:

  • AWS SDK for Java – There are two different ways to include the SDK for Java in your project. You can either download and install the SDK for Java, or if you use Maven to obtain the Amazon SQS Extended Client Library for Java (the SDK for Java is included as a dependency). The SDK for Java and Amazon SQS Extended Client Library for Java require the J2SE Development Kit 8.0 or later. For information about downloading the SDK for Java, see SDK for Java.

    The Amazon SQS Extended Client Library for Java includes support for Maven:

    <dependency> <groupId>com.amazonaws</groupId> <artifactId>amazon-sqs-java-extended-client-lib</artifactId> <version>1.0.1</version> </dependency>
  • Amazon SQS Extended Client Library for Java – If you don't use Maven, you must add package file amazon-sqs-java-extended-client-lib.jar to the Java build class path. For information about downloading the library, see Amazon SQS Extended Client Library for Java.

  • Amazon S3 bucket – You must create a new Amazon S3 bucket or use an existing bucket to store messages. We recommend that you create a new bucket. To control bucket space and charges to your AWS account, you should also set a lifecycle configuration rule on the bucket to permanently delete message objects after a certain period of time after their creation date. For more information, see Managing Lifecycle Configuration or the following example.

Getting Started the Amazon SQS Extended Client Library for Java

The following example code creates an Amazon S3 bucket with a random name and adds a lifecycle rule to permanently delete objects after 14 days. Next, it creates a queue and sends a random message more than 256 KB in size to the queue. The message is stored in the Amazon S3 bucket. The code then consumes the message and returns information about the message. Finally, the message, queue, and bucket are deleted.

/* * Copyright 2010-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"). * You may not use this file except in compliance with the License. * A copy of the License is located at * * http://aws.amazon.com/apache2.0 * * or in the "license" file accompanying this file. This file is distributed * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either * express or implied. See the License for the specific language governing * permissions and limitations under the License. */ import java.util.Arrays; import java.util.List; import java.util.UUID; import com.amazon.sqs.javamessaging.AmazonSQSExtendedClient; import com.amazonaws.services.sqs.AmazonSQS; import com.amazonaws.services.sqs.AmazonSQSClientBuilder; import org.joda.time.DateTime; import org.joda.time.format.DateTimeFormat; import com.amazonaws.services.s3.AmazonS3; import com.amazonaws.services.s3.AmazonS3ClientBuilder; import com.amazonaws.services.s3.model.BucketLifecycleConfiguration; import com.amazonaws.services.s3.model.ListVersionsRequest; import com.amazonaws.services.s3.model.ObjectListing; import com.amazonaws.services.s3.model.S3ObjectSummary; import com.amazonaws.services.s3.model.S3VersionSummary; import com.amazonaws.services.s3.model.VersionListing; import com.amazonaws.services.sqs.model.CreateQueueRequest; import com.amazonaws.services.sqs.model.DeleteMessageRequest; import com.amazonaws.services.sqs.model.DeleteQueueRequest; import com.amazonaws.services.sqs.model.Message; import com.amazonaws.services.sqs.model.ReceiveMessageRequest; import com.amazonaws.services.sqs.model.SendMessageRequest; import com.amazon.sqs.javamessaging.ExtendedClientConfiguration; public class SQSExtendedClientExample { private static final String s3BucketName = UUID.randomUUID() + "-" + DateTimeFormat.forPattern("yyMMdd-hhmmss").print(new DateTime()); public static void main(String[] args) { // Create a new instance of the builder with all defaults (such as credentials and region) set AmazonS3 s3 = AmazonS3ClientBuilder.defaultClient(); // Set the Amazon S3 bucket name, and set a lifecycle rule on the bucket to // permanently delete objects a certain number of days after each object's creation date. // Next, create the bucket, and enable message objects to be stored in the bucket. BucketLifecycleConfiguration.Rule expirationRule = new BucketLifecycleConfiguration.Rule(); expirationRule.withExpirationInDays(14).withStatus("Enabled"); BucketLifecycleConfiguration lifecycleConfig = new BucketLifecycleConfiguration().withRules(expirationRule); s3.createBucket(s3BucketName); s3.setBucketLifecycleConfiguration(s3BucketName, lifecycleConfig); System.out.println("Bucket created and configured."); // Set the Amazon SQS extended client configuration with large payload support enabled. ExtendedClientConfiguration extendedClientConfig = new ExtendedClientConfiguration() .withLargePayloadSupportEnabled(s3, s3BucketName); AmazonSQS sqsExtended = new AmazonSQSExtendedClient(AmazonSQSClientBuilder.defaultClient(), extendedClientConfig); // Create a long string of characters for the message object to be stored in the bucket. int stringLength = 300000; char[] chars = new char[stringLength]; Arrays.fill(chars, 'x'); String myLongString = new String(chars); // Create a message queue for this example. String QueueName = "QueueName" + UUID.randomUUID().toString(); CreateQueueRequest createQueueRequest = new CreateQueueRequest(QueueName); String myQueueUrl = sqsExtended.createQueue(createQueueRequest).getQueueUrl(); System.out.println("Queue created."); // Send the message. SendMessageRequest myMessageRequest = new SendMessageRequest(myQueueUrl, myLongString); sqsExtended.sendMessage(myMessageRequest); System.out.println("Sent the message."); // Receive messages, and then return information about them. ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(myQueueUrl); List<Message> messages = sqsExtended.receiveMessage(receiveMessageRequest).getMessages(); for (Message message : messages) { System.out.println("\nMessage received:"); System.out.println(" ID: " + message.getMessageId()); System.out.println(" Receipt handle: " + message.getReceiptHandle()); System.out.println(" Message body (first 5 characters): " + message.getBody().substring(0, 5)); } // Delete the message, the queue, and the bucket. String messageReceiptHandle = messages.get(0).getReceiptHandle(); sqsExtended.deleteMessage(new DeleteMessageRequest(myQueueUrl, messageReceiptHandle)); System.out.println("Deleted the message."); sqsExtended.deleteQueue(new DeleteQueueRequest(myQueueUrl)); System.out.println("Deleted the queue."); deleteBucketAndAllContents(s3); System.out.println("Deleted the bucket."); } private static void deleteBucketAndAllContents(AmazonS3 client) { ObjectListing objectListing = client.listObjects(s3BucketName); while (true) { for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) { client.deleteObject(s3BucketName, objectSummary.getKey()); } if (objectListing.isTruncated()) { objectListing = client.listNextBatchOfObjects(objectListing); } else { break; } } VersionListing list = client.listVersions(new ListVersionsRequest().withBucketName(s3BucketName)); for (S3VersionSummary s : list.getVersionSummaries()) { client.deleteVersion(s3BucketName, s.getKey(), s.getVersionId()); } client.deleteBucket(s3BucketName); } }