Batch-Based Architecture
AWS Batch
With AWS Batch, you package your application in a container, specify your job’s dependencies, and submit your batch jobs using the AWS Management Console, the CLI, or an SDK. You can specify execution parameters and job dependencies and integrate with a broad range of popular batch computing workflow engines and languages (for example, Pegasus WMS, Luigi, and AWS Step Functions). AWS Batch provides default job queues and compute environment definitions that enable you to get started quickly.
An AWS Batch based architecture can be used for both loosely and tightly coupled workloads. Tightly coupled workloads should use Multi-node Parallel Jobs in AWS Batch.
Reference Architecture
Figure 2: Example AWS Batch architecture
Workflow steps:
-
User creates a job container, uploads the container to the Amazon Elastic Container Registry or another container registry (for example, DockerHub), and creates a job definition to AWS Batch.
-
User submits jobs to a job queue in AWS Batch.
-
AWS Batch pulls the image from the container registry and processes the jobs in the queue
-
Input and output data from each job is stored in an S3 bucket.