使用 Java 和 Amazon S3 管理大型 Amazon SQS 消息 - Amazon Simple Queue Service

本文属于机器翻译版本。若本译文内容与英语原文存在差异,则一律以英文原文为准。

使用 Java 和 Amazon S3 管理大型 Amazon SQS 消息

使用带有 Amazon S3 的 Java 版 Amazon SQS 扩展客户端库来管理大型亚马逊 SQS 消息,尤其适用于介于 256 KB 到 2 GB 之间的有效负载。该库将消息负载存储在 Amazon S3 存储桶中,并发送一条包含对 Amazon SQS 队列中存储对象的引用的消息。

使用适用于 Java 的 Amazon SQS 扩展客户端库,您可以:

  • 指定消息是始终存储在 Amazon S3 中还是仅在其大小超过 256 KB 时存储在 Amazon S3 中。

  • 发送引用存储在 S3 存储桶中的单个消息对象的消息

  • 从 Amazon S3 存储桶检索消息对象

  • 从 Amazon S3 存储桶中删除消息对象

先决条件

以下示例使用 AWS Java 开发工具包。要安装和设置 SDK,请参阅《AWS SDK for Java 开发者指南》中的 “设置 AWS 适用于 Java 的 SDK”。

在运行示例代码之前,请配置您的 AWS 证书。有关更多信息,请参阅《AWS SDK for Java 开发人员指南》中的设置开发 AWS 凭证和区域

适用于 Java 的 SDK 和适用于 Java 的 Amazon SQS 扩展型客户端库需要使用 J2SE Development Kit 8.0 或更高版本。

注意

您可以通过适用于 Java 的 Amazon SQS 扩展型客户端库,利用 Amazon S3 与 AWS SDK for Java来管理 Amazon SQS 消息。使用亚马逊 SQS 控制台 AWS CLI、亚马逊 SQS HTTP API 或任何其他控制台都无法执行此操作。 AWS SDKs

AWS SDK for Java 2.x 示例:使用 Amazon S3 管理大型亚马逊 SQS 消息

以下适用于 Java 的 SDK 2.x 示例创建具有随机名称的 Amazon S3 存储桶并添加生命周期规则以便在创建日期之后的 14 天后永久删除这些对象。它还会创建一个名为 MyQueue 的队列并向该队列发送一条存储在 S3 存储桶中且大小超过 256KB 的随机消息。最后,代码会检索该消息,返回该消息的相关信息,并删除该消息、队列和存储桶。

/* * Copyright 2010-2024 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"). * You may not use this file except in compliance with the License. * A copy of the License is located at * * https://aws.amazon.com/apache2.0 * * or in the "license" file accompanying this file. This file is distributed * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either * express or implied. See the License for the specific language governing * permissions and limitations under the License. * */ import com.amazon.sqs.javamessaging.AmazonSQSExtendedClient; import com.amazon.sqs.javamessaging.ExtendedClientConfiguration; import org.joda.time.DateTime; import org.joda.time.format.DateTimeFormat; import software.amazon.awssdk.services.s3.S3Client; import software.amazon.awssdk.services.s3.model.BucketLifecycleConfiguration; import software.amazon.awssdk.services.s3.model.CreateBucketRequest; import software.amazon.awssdk.services.s3.model.DeleteBucketRequest; import software.amazon.awssdk.services.s3.model.DeleteObjectRequest; import software.amazon.awssdk.services.s3.model.ExpirationStatus; import software.amazon.awssdk.services.s3.model.LifecycleExpiration; import software.amazon.awssdk.services.s3.model.LifecycleRule; import software.amazon.awssdk.services.s3.model.LifecycleRuleFilter; import software.amazon.awssdk.services.s3.model.ListObjectVersionsRequest; import software.amazon.awssdk.services.s3.model.ListObjectVersionsResponse; import software.amazon.awssdk.services.s3.model.ListObjectsV2Request; import software.amazon.awssdk.services.s3.model.ListObjectsV2Response; import software.amazon.awssdk.services.s3.model.PutBucketLifecycleConfigurationRequest; import software.amazon.awssdk.services.sqs.SqsClient; import software.amazon.awssdk.services.sqs.model.CreateQueueRequest; import software.amazon.awssdk.services.sqs.model.CreateQueueResponse; import software.amazon.awssdk.services.sqs.model.DeleteMessageRequest; import software.amazon.awssdk.services.sqs.model.DeleteQueueRequest; import software.amazon.awssdk.services.sqs.model.Message; import software.amazon.awssdk.services.sqs.model.ReceiveMessageRequest; import software.amazon.awssdk.services.sqs.model.ReceiveMessageResponse; import software.amazon.awssdk.services.sqs.model.SendMessageRequest; import java.util.Arrays; import java.util.List; import java.util.UUID; /** * Examples of using Amazon SQS Extended Client Library for Java 2.x * */ public class SqsExtendedClientExamples { // Create an Amazon S3 bucket with a random name. private final static String S3_BUCKET_NAME = UUID.randomUUID() + "-" + DateTimeFormat.forPattern("yyMMdd-hhmmss").print(new DateTime()); public static void main(String[] args) { /* * Create a new instance of the builder with all defaults (credentials * and region) set automatically. For more information, see * Creating Service Clients in the AWS SDK for Java Developer Guide. */ final S3Client s3 = S3Client.create(); /* * Set the Amazon S3 bucket name, and then set a lifecycle rule on the * bucket to permanently delete objects 14 days after each object's * creation date. */ final LifecycleRule lifeCycleRule = LifecycleRule.builder() .expiration(LifecycleExpiration.builder().days(14).build()) .filter(LifecycleRuleFilter.builder().prefix("").build()) .status(ExpirationStatus.ENABLED) .build(); final BucketLifecycleConfiguration lifecycleConfig = BucketLifecycleConfiguration.builder() .rules(lifeCycleRule) .build(); // Create the bucket and configure it s3.createBucket(CreateBucketRequest.builder().bucket(S3_BUCKET_NAME).build()); s3.putBucketLifecycleConfiguration(PutBucketLifecycleConfigurationRequest.builder() .bucket(S3_BUCKET_NAME) .lifecycleConfiguration(lifecycleConfig) .build()); System.out.println("Bucket created and configured."); // Set the Amazon SQS extended client configuration with large payload support enabled final ExtendedClientConfiguration extendedClientConfig = new ExtendedClientConfiguration().withPayloadSupportEnabled(s3, S3_BUCKET_NAME); final SqsClient sqsExtended = new AmazonSQSExtendedClient(SqsClient.builder().build(), extendedClientConfig); // Create a long string of characters for the message object int stringLength = 300000; char[] chars = new char[stringLength]; Arrays.fill(chars, 'x'); final String myLongString = new String(chars); // Create a message queue for this example final String queueName = "MyQueue-" + UUID.randomUUID(); final CreateQueueResponse createQueueResponse = sqsExtended.createQueue(CreateQueueRequest.builder().queueName(queueName).build()); final String myQueueUrl = createQueueResponse.queueUrl(); System.out.println("Queue created."); // Send the message final SendMessageRequest sendMessageRequest = SendMessageRequest.builder() .queueUrl(myQueueUrl) .messageBody(myLongString) .build(); sqsExtended.sendMessage(sendMessageRequest); System.out.println("Sent the message."); // Receive the message final ReceiveMessageResponse receiveMessageResponse = sqsExtended.receiveMessage(ReceiveMessageRequest.builder().queueUrl(myQueueUrl).build()); List<Message> messages = receiveMessageResponse.messages(); // Print information about the message for (Message message : messages) { System.out.println("\nMessage received."); System.out.println(" ID: " + message.messageId()); System.out.println(" Receipt handle: " + message.receiptHandle()); System.out.println(" Message body (first 5 characters): " + message.body().substring(0, 5)); } // Delete the message, the queue, and the bucket final String messageReceiptHandle = messages.get(0).receiptHandle(); sqsExtended.deleteMessage(DeleteMessageRequest.builder().queueUrl(myQueueUrl).receiptHandle(messageReceiptHandle).build()); System.out.println("Deleted the message."); sqsExtended.deleteQueue(DeleteQueueRequest.builder().queueUrl(myQueueUrl).build()); System.out.println("Deleted the queue."); deleteBucketAndAllContents(s3); System.out.println("Deleted the bucket."); } private static void deleteBucketAndAllContents(S3Client client) { ListObjectsV2Response listObjectsResponse = client.listObjectsV2(ListObjectsV2Request.builder().bucket(S3_BUCKET_NAME).build()); listObjectsResponse.contents().forEach(object -> { client.deleteObject(DeleteObjectRequest.builder().bucket(S3_BUCKET_NAME).key(object.key()).build()); }); ListObjectVersionsResponse listVersionsResponse = client.listObjectVersions(ListObjectVersionsRequest.builder().bucket(S3_BUCKET_NAME).build()); listVersionsResponse.versions().forEach(version -> { client.deleteObject(DeleteObjectRequest.builder().bucket(S3_BUCKET_NAME).key(version.key()).versionId(version.versionId()).build()); }); client.deleteBucket(DeleteBucketRequest.builder().bucket(S3_BUCKET_NAME).build()); } }

您可以使用 Apache Maven 配置和构建适用于 Java 项目的 Amazon SQS 扩展型客户端或构建 SDK 本身。指定您的应用程序中使用的 SDK 的各个模块。

<properties> <aws-java-sdk.version>2.20.153</aws-java-sdk.version> </properties> <dependencies> <dependency> <groupId>software.amazon.awssdk</groupId> <artifactId>sqs</artifactId> <version>${aws-java-sdk.version}</version> </dependency> <dependency> <groupId>software.amazon.awssdk</groupId> <artifactId>s3</artifactId> <version>${aws-java-sdk.version}</version> </dependency> <dependency> <groupId>com.amazonaws</groupId> <artifactId>amazon-sqs-java-extended-client-lib</artifactId> <version>2.0.4</version> </dependency> <dependency> <groupId>joda-time</groupId> <artifactId>joda-time</artifactId> <version>2.12.6</version> </dependency> </dependencies>