What Is AWS Batch? - AWS Batch

What Is AWS Batch?

AWS Batch helps you to run batch computing workloads on the AWS Cloud. Batch computing is a common way for developers, scientists, and engineers to access large amounts of compute resources. AWS Batch removes the undifferentiated heavy lifting of configuring and managing the required infrastructure, similar to traditional batch computing software. This service can efficiently provision resources in response to jobs submitted in order to eliminate capacity constraints, reduce compute costs, and deliver results quickly.

As a fully managed service, AWS Batch helps you to run batch computing workloads of any scale. AWS Batch automatically provisions compute resources and optimizes the workload distribution based on the quantity and scale of the workloads. With AWS Batch, there's no need to install or manage batch computing software, so you can focus your time on analyzing results and solving problems.

Components of AWS Batch

AWS Batch simplifies running batch jobs across multiple Availability Zones within a Region. You can create AWS Batch compute environments within a new or existing VPC. After a compute environment is up and associated with a job queue, you can define job definitions that specify which Docker container images to run your jobs. Container images are stored in and pulled from container registries, which may exist within or outside of your AWS infrastructure.

Jobs

A unit of work (such as a shell script, a Linux executable, or a Docker container image) that you submit to AWS Batch. It has a name, and runs as a containerized application on AWS Fargate or Amazon EC2 resources in your compute environment, using parameters that you specify in a job definition. Jobs can reference other jobs by name or by ID, and can be dependent on the successful completion of other jobs. For more information, see Jobs.

Job Definitions

A job definition specifies how jobs are to be run. You can think of a job definition as a blueprint for the resources in your job. You can supply your job with an IAM role to provide access to other AWS resources. You also specify both memory and CPU requirements. The job definition can also control container properties, environment variables, and mount points for persistent storage. Many of the specifications in a job definition can be overridden by specifying new values when submitting individual Jobs. For more information, see Job definitions

Job Queues

When you submit an AWS Batch job, you submit it to a particular job queue, where the job resides until it's scheduled onto a compute environment. You associate one or more compute environments with a job queue. You can also assign priority values for these compute environments and even across job queues themselves. For example, you can have a high priority queue that you submit time-sensitive jobs to, and a low priority queue for jobs that can run anytime when compute resources are cheaper.

Compute Environment

A compute environment is a set of managed or unmanaged compute resources that are used to run jobs. With managed compute environments, you can specify desired compute type (Fargate or EC2) at several levels of detail. You can set up compute environments that use a particular type of EC2 instance, a particular model such as c5.2xlarge or m5.10xlarge. Or, you can choose only to specify that you want to use the newest instance types. You can also specify the minimum, desired, and maximum number of vCPUs for the environment, along with the amount that you're willing to pay for a Spot Instance as a percentage of the On-Demand Instance price and a target set of VPC subnets. AWS Batch efficiently launches, manages, and terminates compute types as needed. You can also manage your own compute environments. As such, you're responsible for setting up and scaling the instances in an Amazon ECS cluster that AWS Batch creates for you. For more information, see Compute environment.

Getting Started

Get started with AWS Batch by creating a job definition, compute environment, and a job queue in the AWS Batch console.

The AWS Batch first-run wizard gives you the option of creating a compute environment and a job queue and submitting a sample Hello World job. If you already have a Docker image you want to launch in AWS Batch, you can create a job definition with that image and submit that to your queue instead. For more information, see Getting Started with AWS Batch.

Dashboard

On the AWS Batch dashboard, you can monitor recent jobs, job queues, and compute environments. By default, the following dashboard widgets are displayed:

  • Job overview – For more information about AWS Batch jobs, see Jobs.

  • Job queue overview – For more information about AWS Batch job queues, see Job queues.

  • Compute environment overview – For more information about AWS Batch compute environments, see Compute environment.

You can customize the widgets that are displayed on the Dashboard page. The following sections describe additional widgets that you can install.

Single job queue

This widget displays detailed information about a single job queue.

To add this widget, follow these steps.

  1. Open the AWS Batch console.

  2. From the navigation bar, select the AWS Region that you want.

  3. In the navigation pane, choose Dashboard.

  4. Choose Add widgets.

  5. For Single job queue, choose Add widget.

  6. For Job queue, select the job queue that you want.

  7. For Job status, choose the job statuses that you want to display.

  8. (Optional) Turn off Show connected compute environments if you don't want to display the properties for compute environments.

  9. For Compute environment properties, select the properties that you want.

  10. Choose Add.

CloudWatch Container Insights

This widget displays aggregated metrics for AWS Batch compute environments and jobs. For more information about Container Insights, see AWS Batch CloudWatch Container Insights.

To add this widget, follow these steps.

  1. Open the AWS Batch console.

  2. From the navigation bar, select the AWS Region that you want.

  3. In the navigation pane, choose Dashboard.

  4. Choose Add widgets.

  5. For Container insights, choose Add widget.

  6. For Compute environment, choose the compute environment that you want.

  7. Choose Add.

Job logs

This widget displays different logs from your jobs in one convenient location. For more information about job logs, see Job logs.

To add this widget, follow these steps.

  1. Open the AWS Batch console.

  2. From the navigation bar, select the AWS Region that you want.

  3. In the navigation pane, choose Dashboard.

  4. Choose Add widgets.

  5. For Job logs, choose Add widget.

  6. For Job id, enter the job ID for the job that you want.

  7. Choose Add.