Run event-driven and scheduled workloads at scale with AWS Fargate
Created by HARI OHM PRASATH RAJAGOPAL (AWS)
Environment: PoC or pilot | Technologies: Modernization; Serverless; Operations | Workload: Open-source |
AWS services: Amazon EC2 Container Registry; Amazon ECS; AWS CodeCommit; AWS Fargate; AWS Lambda; Amazon SNS |
Summary
This pattern describes how to run scheduled and event-driven workloads at scale on the Amazon Web Services (AWS) Cloud by using AWS Fargate.
In the use case that this pattern sets up, code is scanned for AWS sensitive information, such as the AWS account number and credentials, whenever a pull request is submitted. The pull request initiates a Lambda function. The Lambda function invokes a Fargate task that takes care of the code scan. Lambda is initiated whenever a new pull request is raised. If the scan finds any sensitive information, Amazon Simple Notification Service (Amazon SNS) sends the scan results in an email message.
This pattern is helpful in the following business use cases:
If your business must run many scheduled and event-driven workloads that cannot be run by AWS Lambda because of limitations around runtime (a 15-minute limit) or memory
If you want AWS to manage the instances provisioned for these workloads
When you use this pattern, you have the option of creating a new virtual private cloud (VPC).
Prerequisites and limitations
Prerequisites
An active AWS account
AWS CodeCommit for hosting the code base and creating pull requests
AWS Command Line Interface (AWS CLI) version 1.7 or later, installed and configured on macOS, Linux, or Windows
Workloads running in containers
Apache Maven executable set up in classpath
Architecture
The overall flow includes the following steps.
Whenever a new pull request is submitted in CodeCommit, a Lambda function is initiated. The Lambda function listens through the
CodeCommit Pull Request State Changeevent through Amazon EventBridge.The Lambda function submits a new Fargate task with the following environment parameters for checking out the code and scanning it.
RUNNER → <<TaskARN>> SNS_TOPIC → <<SNSTopicARN>> SUBNET → <<Subnet in which Fargate task gets launched>>If the scan finds sensitive information in the code, Fargate pushes a new message to the Amazon SNS topic.
An SNS subscriber reads the message from the topic and sends an email message.
Technology
AWS CodeCommit
Amazon Elastic Container Registry (Amazon ECR)
Amazon Elastic Container Service (Amazon ECS)
Amazon EventBridge
AWS Fargate
AWS Lambda
Amazon SNS
Docker
Tools
Tools
AWS CLI – AWS Command Line Interface (CLI) is a unified tool to manage your AWS services.
AWS CodeCommit
– AWS CodeCommit is a fully managed source control service that hosts secure Git-based repositories. Using CodeCommit, teams can collaborate on code in a secure and highly scalable environment. Amazon ECR – Amazon Elastic Container Registry (Amazon ECR) is a fully managed registry that developers can use to store, manage, and deploy Docker container images.
Amazon ECS – Amazon Elastic Container Service (Amazon ECS) is a highly scalable, fast container management service. You can use Amazon ECS to run, stop, and manage containers on a cluster.
AWS Fargate – AWS Fargate is a technology that you can use with Amazon ECS to run containers without having to manage servers or clusters of Amazon EC2 instances.
AWS Lambda – AWS Lambda is a compute service that supports running code without provisioning or managing servers. Lambda runs your code only when needed and scales automatically, from a few requests per day to thousands per second.
Amazon SNS – Amazon Simple Notification Service (Amazon SNS) is a managed service that provides message delivery from publishers to subscribers (also known as producers and consumers). Publishers communicate asynchronously with subscribers by sending messages to a topic, which is a logical access point and communication channel. Clients that subscribe to the SNS topic receive published messages using a supported protocol, such Lambda, email, mobile push notifications, and mobile text messages (SMS).
Docker
– Docker helps you build, test, and deliver applications in packages called containers. Git client
– Command line or desktop tool to check out the required artifacts Maven
– Apache Maven is a project management tool for centrally managing a project’s build, reporting, and documentation.
Epics
| Task | Description | Skills required |
|---|---|---|
Download the code. | In the Attachments section, download the .zip file and extract the files. | Developer, AWS system administrator |
Set up the repo. | Run | Developer, AWS system administrator |
| Task | Description | Skills required |
|---|---|---|
Create an Amazon ECR repository and log in. | Open the Amazon ECR console. In the navigation pane, choose Repositories, and then choose Create repository. For help with this and other stories, see the Related resources section. | Developer, AWS system administrator |
Push your container image. | Open the repository, choose View push commands, and log in to Docker. After you are logged in, run the commands, with the required substitutions, that are under Push the container image in the Additional information section. This uploads the Docker container image that is used to perform code scanning. After the upload is complete, copy the URL of the latest build in the Amazon ECR repository. | Developer, AWS system administrator |
| Task | Description | Skills required |
|---|---|---|
Create the CodeCommit repository. | To create a new AWS CodeCommit repository, run the command under Create the CodeCommit repository in the Additional information section. | Developer, AWS system administrator |
| Task | Description | Skills required |
|---|---|---|
Create a VPC. | If you want to use a new VPC rather than an existing one, run the commands under Create a VPC in the Additional information section. The AWS Cloud Development Kit (AWS CDK) script will output the IDs of the VPC and subnet that were created. | Developer, AWS system administrator |
| Task | Description | Skills required |
|---|---|---|
Create the cluster and the task. | To create an Amazon ECS cluster and Fargate task definition, run the commands under Create the cluster and task in the Additional information section. Make sure that the correct VPC ID and Amazon ECR repo URI are passed in as a parameter while running the shell script. The script creates a Fargate task definition that points to the Docker image (responsible for scanning). The script then creates a job and an associated execution role. | Developer, AWS system administrator |
Verify the Amazon ECS cluster. | Open the Amazon ECS console. In the navigation pane, choose Clusters, and choose the newly created Amazon ECS cluster named Fargate-Job-Cluster. After this, choose Task definition in the navigation pane, and confirm that there is a new task definition with the prefix | Developer, AWS system administrator |
| Task | Description | Skills required |
|---|---|---|
Create an SNS topic. | To create an SNS topic, run the command under Create the SNS topic in the Additional information section. After creation is successful, note the | Developer, AWS system administrator |
Create the SNS subscriber. | To create an email subscriber for the SNS topic, run the command under Create the SNS subscriber in the Additional information section. Make sure to replace | Developer, AWS system administrator |
| Task | Description | Skills required |
|---|---|---|
Create the function and the trigger. | To create a Lambda function with a CodeCommit trigger, run the command under Lambda function and CodeCommit trigger in the Additional information section. Make sure to replace the parameters with the corresponding values before running the command. The script creates the Lambda function and configures it to be invoked when a new pull request is made. | Developer, AWS system administrator |
| Task | Description | Skills required |
|---|---|---|
Test the application. | If you check in any AWS sensitive information to CodeCommit repo, the Lambda function should be initiated. The Lambda function initiates the Fargate task, which scans the code and sends the scan results in an email notification. | Developer, AWS system administrator |
Related resources
Additional information
Push the container image
> cd 1-ecr-image-push > ./run.sh <<ecr-repository>>
Create the CodeCommit repository
aws codecommit create-repository --repository-name test-repo --repository-description "My Test repository"
Create a VPC
> cd 2-create-vpc > ./run.sh
Output
aws-batch-cdk-vpc-efs-launch-template.privatesubnet = subnet-<<id>> aws-batch-cdk-vpc-efs-launch-template.publicsubnet = subnet-<<id>> aws-batch-cdk-vpc-efs-launch-template.vpcid = vpc-<<id>>
Create the cluster and task
> export CDK_DEFAULT_ACCOUNT = <<aws_account_id>> > export CDK_DEFAULT_REGION = <<aws_region>> > cd 3-create-ecs-task > ./run.sh <<vpc-id>> <<ecr-repo-uri>>
Output
aws-cdk-fargate-ecs.CLUSTERNAME = Fargate-Job-Cluster aws-cdk-fargate-ecs.ClusterARN = <<cluster_arn>> aws-cdk-fargate-ecs.ContainerARN = Fargate-Container aws-cdk-fargate-ecs.TaskARN = <<task_arn>> aws-cdk-fargate-ecs.TaskExecutionRole = <<execution_role_arn>> aws-cdk-fargate-ecs.TaskRole = <<task_role_arn>>
Create the SNS topic
aws sns create-topic --name code-commit-topic
Create the SNS subscriber
aws sns subscribe \ --topic-arn <<topic_arn>> \ --protocol email \ --notification-endpoint <<email_address>>
Lambda function and CodeCommit trigger
> export CDK_DEFAULT_ACCOUNT = <<aws_account_id>> > export CDK_DEFAULT_REGION = <<aws_region>> > cd 5-Lambda-CodeCommit-Trigger > ./run.sh <<taskarn>> <<snstopicarn>> subnet-<<id>> <<codecommitarn>>
Output
aws-cdk-fargate-lambda-event.Cloudwatchrule = <<cloudwatchrule>> aws-cdk-fargate-lambda-event.CodeCommitLambda = AWS-Code-Scanner-Function aws-cdk-fargate-lambda-event.LambdaRole = <<lambdaiamrole>>
Attachments
To access additional content that is associated with this document, unzip the following file: attachment.zip
