Run message-driven workloads at scale by using AWS Fargate - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsEpicsRelated resources

Run message-driven workloads at scale by using AWS Fargate

Created by Stan Zubarev (AWS)

Environment: PoC or pilot

Technologies: Containers & microservices; Messaging & communications; Databases

AWS services: AWS Fargate; Amazon SQS; Amazon DynamoDB

Summary

This pattern shows how to run message-driven workloads at scale in the AWS Cloud by using containers and AWS Fargate.

Using containers to process data can be helpful when the amount of data an application processes exceeds the limitations of function-based serverless compute services. For example, if an application requires more compute capacity or processing time than what AWS Lambda offers, using Fargate can improve performance.

The following example setup uses the AWS Cloud Development Kit (AWS CDK) in TypeScript to configure and deploy the following resources in the AWS Cloud:

  • A Fargate service

  • An Amazon Simple Queue Service (Amazon SQS) queue

  • An Amazon DynamoDB table.

  • An Amazon CloudWatch dashboard

The Fargate service receives and processes messages from the Amazon SQS queue, then stores them in the Amazon DynamoDB table. You can monitor how many Amazon SQS messages are processed and how many DynamoDB items are created by Fargate by using the CloudWatch dashboard.

Note: You can also use this pattern’s example code to build more complex data processing workloads in event-driven serverless architectures. For more information, see Run event-driven and scheduled workloads at scale with AWS Fargate.

Prerequisites and limitations

Prerequisites 

  • An active AWS account

  • The latest version of the AWS Command Line Interface (AWS CLI), installed and configured on your local machine

  • Git, installed and configured on your local machine

  • The AWS CDK, installed and configured on your local machine

  • Go, installed and configured on your local machine

  • Docker, installed and configured on your local machine

Architecture

Target technology stack  

  • Amazon SQS

  • AWS Fargate

  • Amazon DynamoDB

Target architecture 

The following diagram shows an example workflow for running message-driven workloads at scale in the AWS Cloud by using Fargate:

Example workflow for message-driven workloads

The diagram shows the following workflow:

  1. The Fargate service uses Amazon SQS long polling to receive messages from an Amazon SQS queue.

  2. The Fargate service then processes the Amazon SQS messages and stores them in a DynamoDB table.

Automation and scale

To automate scaling your Fargate task count, you can configure Amazon Elastic Container Service (Amazon ECS) Service Auto Scaling. It’s a best practice to configure the scaling policy based on the number of visible messages in your application’s Amazon SQS queue.

For more information, see Scaling based on Amazon SQS in the Amazon EC2 Auto Scaling User Guide.

Tools

AWS services

  • AWS Fargate helps you run containers without needing to manage servers or Amazon Elastic Compute Cloud (Amazon EC2) instances. It’s used in conjunction with Amazon Elastic Container Service (Amazon ECS).

  • Amazon Simple Queue Service (Amazon SQS) provides a secure, durable, and available hosted queue that helps you integrate and decouple distributed software systems and components.

  • Amazon DynamoDB is a fully managed NoSQL database service that provides fast, predictable, and scalable performance.

  • Amazon CloudWatch helps you monitor the metrics of your AWS resources and the applications you run on AWS in real time.

Code 

The code for this pattern is available in the GitHub sqs-fargate-ddb-cdk-go repository.

Epics

TaskDescriptionSkills required

Clone the GitHub repository.

Clone the GitHub sqs-fargate-ddb-cdk-go repository to your local machine by running the following command:

git clone https://github.com/aws-samples/sqs-fargate-ddb-cdk-go.git

App developer

Verify that the AWS CLI is configured to the correct AWS account and that the AWS CDK has the required permissions.

To check if your AWS CLI configuration settings are correct, you can run the following Amazon Simple Storage Service (Amazon S3) ls command:

aws s3 ls

This procedure also requires the AWS CDK to have permissions to provision infrastructure within your AWS account. To grant the required permissions, you must create named AWS profile in AWS CLI and export it as an AWS_PROFILE environment variable.

Note: If you haven’t used the AWS CDK in your AWS account before, you must first provision the required AWS CDK resources. For more information, see Bootstrapping in the AWS CDK v2 Developer Guide.

App developer

Deploy the AWS CDK stack to your AWS account.

  1. Build a container image by running the following AWS CLI command:

    docker build -t go-fargate .

  2. Open the AWS CDK directory by running the following command:

    cd cdk

  3. Install the required npm modules by running the following command:

    npm i

  4. Deploy the AWS CDK pattern to your AWS account by running the following command:

    cdk deploy --profile ${AWS_PROFILE}

App developer
TaskDescriptionSkills required

Send a test message to the Amazon SQS queue.

For instructions, see Sending messages to a queue (console) in the Amazon SQS Developer Guide.

Test Amazon SQS message example

{
    "message": "hello, Fargate"
}
App developer

Verify that the test message appears in the Fargate service's CloudWatch logs.

Follow the instructions in Viewing CloudWatch Logs in the Amazon ECS Developer Guide. Make sure that you review the logs for the go-fargate-service log group in the go-service-cluster ECS cluster.

App developer

Verify that the test message appears in the DynamoDB table.

  1. Open the DynamoDB console.

  2. In the left navigation pane, choose Tables. Then, select the following table from the list: sqs-fargate-ddb-table.

  3. Choose Explore table items.

  4. Verify that the test message appears in the Items returned list.

App developer

Verify that the Fargate service is sending messages to CloudWatch Logs.

  1. Open the CloudWatch console.

  2. In the left navigation pane, choose Dashboards.

  3. In the Custom Dashboards list, select the dashboard named go-service-dashboard.

  4. Verify that the test message appears in the logs.

Note: The AWS CDK creates the CloudWatch dashboard in your AWS account automatically.

App developer
TaskDescriptionSkills required

Delete the AWS CDK stack.

  1. Open your AWS CDK directory in the AWS CLI by running the following command:

    cd cdk

  2. Delete the AWS CDK stack by running the following command:

    cdk destroy --profile ${AWS_PROFILE}

App developer

Verify that the AWS CDK stack is deleted.

To make sure that the stack was deleted, run the following command:

aws cloudformation list-stacks --query \ 
"StackSummaries[?contains(StackName,'SqsFargate')].StackStatus" \
--profile ${AWS_PROFILE}

The StackStatus value returned in the command output is DELETE_COMPLETE if the stack is deleted.

For more information, see Describing and listing your stacks in the AWS CloudFormation User Guide.

App developer

Related resources