Interface | Description |
---|---|
CfnComputeEnvironment.ComputeResourcesProperty |
Details about the compute resources managed by the compute environment.
|
CfnComputeEnvironment.Ec2ConfigurationObjectProperty |
Provides information used to select Amazon Machine Images (AMIs) for instances in the compute environment.
|
CfnComputeEnvironment.EksConfigurationProperty |
Configuration for the Amazon EKS cluster that supports the AWS Batch compute environment.
|
CfnComputeEnvironment.LaunchTemplateSpecificationProperty |
An object that represents a launch template that's associated with a compute resource.
|
CfnComputeEnvironment.UpdatePolicyProperty |
Specifies the infrastructure update policy for the compute environment.
|
CfnComputeEnvironmentProps |
Properties for defining a `CfnComputeEnvironment`.
|
CfnJobDefinition.AuthorizationConfigProperty |
The authorization configuration details for the Amazon EFS file system.
|
CfnJobDefinition.ContainerPropertiesProperty |
Container properties are used for Amazon ECS based job definitions.
|
CfnJobDefinition.DeviceProperty |
An object that represents a container instance host device.
|
CfnJobDefinition.EfsVolumeConfigurationProperty |
This is used when you're using an Amazon Elastic File System file system for job storage.
|
CfnJobDefinition.EksContainerEnvironmentVariableProperty |
An environment variable.
|
CfnJobDefinition.EksContainerProperty |
EKS container properties are used in job definitions for Amazon EKS based job definitions to describe the properties for a container node in the pod that's launched as part of a job.
|
CfnJobDefinition.EksContainerVolumeMountProperty |
The volume mounts for a container for an Amazon EKS job.
|
CfnJobDefinition.EksPropertiesProperty |
An object that contains the properties for the Kubernetes resources of a job.
|
CfnJobDefinition.EksVolumeProperty |
Specifies an Amazon EKS volume for a job definition.
|
CfnJobDefinition.EmptyDirProperty |
Specifies the configuration of a Kubernetes `emptyDir` volume.
|
CfnJobDefinition.EnvironmentProperty |
The Environment property type specifies environment variables to use in a job definition.
|
CfnJobDefinition.EvaluateOnExitProperty |
Specifies an array of up to 5 conditions to be met, and an action to take ( `RETRY` or `EXIT` ) if all conditions are met.
|
CfnJobDefinition.FargatePlatformConfigurationProperty |
The platform configuration for jobs that are running on Fargate resources.
|
CfnJobDefinition.HostPathProperty |
Specifies the configuration of a Kubernetes `hostPath` volume.
|
CfnJobDefinition.LinuxParametersProperty |
Linux-specific modifications that are applied to the container, such as details for device mappings.
|
CfnJobDefinition.LogConfigurationProperty |
Log configuration options to send to a custom log driver for the container.
|
CfnJobDefinition.MountPointsProperty |
Details for a Docker volume mount point that's used in a job's container properties.
|
CfnJobDefinition.NetworkConfigurationProperty |
The network configuration for jobs that are running on Fargate resources.
|
CfnJobDefinition.NodePropertiesProperty |
An object that represents the node properties of a multi-node parallel job.
|
CfnJobDefinition.NodeRangePropertyProperty |
An object that represents the properties of the node range for a multi-node parallel job.
|
CfnJobDefinition.PodPropertiesProperty |
The properties for the pod.
|
CfnJobDefinition.ResourceRequirementProperty |
The type and amount of a resource to assign to a container.
|
CfnJobDefinition.ResourcesProperty |
The type and quantity of the resources to request for the container.
|
CfnJobDefinition.RetryStrategyProperty |
The retry strategy that's associated with a job.
|
CfnJobDefinition.SecretProperty |
An object that represents the secret to expose to your container.
|
CfnJobDefinition.SecurityContextProperty |
The security context for a job.
|
CfnJobDefinition.TimeoutProperty |
An object that represents a job timeout configuration.
|
CfnJobDefinition.TmpfsProperty |
The container path, mount options, and size of the `tmpfs` mount.
|
CfnJobDefinition.UlimitProperty |
The `ulimit` settings to pass to the container.
|
CfnJobDefinition.VolumesHostProperty |
Determine whether your data volume persists on the host container instance and where it's stored.
|
CfnJobDefinition.VolumesProperty |
A list of volumes that are associated with the job.
|
CfnJobDefinitionProps |
Properties for defining a `CfnJobDefinition`.
|
CfnJobQueue.ComputeEnvironmentOrderProperty |
The order that compute environments are tried in for job placement within a queue.
|
CfnJobQueueProps |
Properties for defining a `CfnJobQueue`.
|
CfnSchedulingPolicy.FairsharePolicyProperty |
The fair share policy for a scheduling policy.
|
CfnSchedulingPolicy.ShareAttributesProperty |
Specifies the weights for the fair share identifiers for the fair share policy.
|
CfnSchedulingPolicyProps |
Properties for defining a `CfnSchedulingPolicy`.
|
ComputeEnvironmentProps |
(experimental) Properties for creating a new Compute Environment.
|
ComputeResources |
(experimental) Properties for defining the structure of the batch compute cluster.
|
IComputeEnvironment |
(experimental) Properties of a compute environment.
|
IComputeEnvironment.Jsii$Default |
Internal default implementation for
IComputeEnvironment . |
IJobDefinition |
(experimental) An interface representing a job definition - either a new one, created with the CDK, *using the
JobDefinition class, or existing ones, referenced using the JobDefinition.fromJobDefinitionArn method. |
IJobDefinition.Jsii$Default |
Internal default implementation for
IJobDefinition . |
IJobQueue |
(experimental) Properties of a Job Queue.
|
IJobQueue.Jsii$Default |
Internal default implementation for
IJobQueue . |
IMultiNodeProps |
(experimental) Properties for specifying multi-node properties for compute resources.
|
IMultiNodeProps.Jsii$Default |
Internal default implementation for
IMultiNodeProps . |
INodeRangeProps |
(experimental) Properties for a multi-node batch job.
|
INodeRangeProps.Jsii$Default |
Internal default implementation for
INodeRangeProps . |
JobDefinitionContainer |
(experimental) Properties of a job definition container.
|
JobDefinitionProps |
(experimental) Construction properties of the
JobDefinition construct. |
JobQueueComputeEnvironment |
(experimental) Properties for mapping a compute environment to a job queue.
|
JobQueueProps |
(experimental) Properties of a batch job queue.
|
LaunchTemplateSpecification |
(experimental) Launch template property specification.
|
LogConfiguration |
(experimental) Log configuration options to send to a custom log driver for the container.
|
Enum | Description |
---|---|
AllocationStrategy |
(experimental) Properties for how to prepare compute resources that are provisioned for a compute environment.
|
ComputeResourceType |
(experimental) Property to specify if the compute environment uses On-Demand, SpotFleet, Fargate, or Fargate Spot compute resources.
|
LogDriver |
(experimental) The log driver to use for the container.
|
PlatformCapabilities |
(experimental) Platform capabilities.
|
---
All classes with the
Cfn
prefix in this module (CFN Resources) are always stable and safe to use.
The APIs of higher level constructs in this module are experimental and under active development. They are subject to non-backward compatible changes or removal in any future version. These are not subject to the Semantic Versioning model and breaking changes will be announced in the release notes. This means that while you may use them, you may need to update your source code when upgrading to a newer version of this package.
This module is part of the AWS Cloud Development Kit project.
AWS Batch is a batch processing tool for efficiently running hundreds of thousands computing jobs in AWS. Batch can dynamically provision different types of compute resources based on the resource requirements of submitted jobs.
AWS Batch simplifies the planning, scheduling, and executions of your batch workloads across a full range of compute services like Amazon EC2 and Spot Resources.
Batch achieves this by utilizing queue processing of batch job requests. To successfully submit a job for execution, you need the following resources:
For more information on AWS Batch visit the AWS Docs for Batch.
At the core of AWS Batch is the compute environment. All batch jobs are processed within a compute environment, which uses resource like OnDemand/Spot EC2 instances or Fargate.
In MANAGED mode, AWS will handle the provisioning of compute resources to accommodate the demand. Otherwise, in UNMANAGED mode, you will need to manage the provisioning of those resources.
Below is an example of each available type of compute environment:
Vpc vpc; // default is managed ComputeEnvironment awsManagedEnvironment = ComputeEnvironment.Builder.create(this, "AWS-Managed-Compute-Env") .computeResources(ComputeResources.builder() .vpc(vpc) .build()) .build(); ComputeEnvironment customerManagedEnvironment = ComputeEnvironment.Builder.create(this, "Customer-Managed-Compute-Env") .managed(false) .build();
It is possible to have AWS Batch submit spotfleet requests for obtaining compute resources. Below is an example of how this can be done:
Vpc vpc = new Vpc(this, "VPC"); ComputeEnvironment spotEnvironment = ComputeEnvironment.Builder.create(this, "MySpotEnvironment") .computeResources(ComputeResources.builder() .type(ComputeResourceType.SPOT) .bidPercentage(75) // Bids for resources at 75% of the on-demand price .vpc(vpc) .build()) .build();
It is possible to have AWS Batch submit jobs to be run on Fargate compute resources. Below is an example of how this can be done:
Vpc vpc = new Vpc(this, "VPC"); ComputeEnvironment fargateSpotEnvironment = ComputeEnvironment.Builder.create(this, "MyFargateEnvironment") .computeResources(ComputeResources.builder() .type(ComputeResourceType.FARGATE_SPOT) .vpc(vpc) .build()) .build();
AWS Batch uses an allocation strategy to determine what compute resource will efficiently handle incoming job requests. By default, BEST_FIT will pick an available compute instance based on vCPU requirements. If none exist, the job will wait until resources become available. However, with this strategy, you may have jobs waiting in the queue unnecessarily despite having more powerful instances available. Below is an example of how that situation might look like:
Compute Environment: 1. m5.xlarge => 4 vCPU 2. m5.2xlarge => 8 vCPU
Job Queue: --------- | A | B | --------- Job Requirements: A => 4 vCPU - ALLOCATED TO m5.xlarge B => 2 vCPU - WAITING
In this situation, Batch will allocate Job A to compute resource #1 because it is the most cost efficient resource that matches the vCPU requirement. However, with this BEST_FIT
strategy, Job B will not be allocated to our other available compute resource even though it is strong enough to handle it. Instead, it will wait until the first job is finished processing or wait a similar m5.xlarge
resource to be provisioned.
The alternative would be to use the BEST_FIT_PROGRESSIVE
strategy in order for the remaining job to be handled in larger containers regardless of vCPU requirement and costs.
Simply define your Launch Template:
// This example is only available in TypeScript const myLaunchTemplate = new ec2.CfnLaunchTemplate(this, 'LaunchTemplate', { launchTemplateName: 'extra-storage-template', launchTemplateData: { blockDeviceMappings: [ { deviceName: '/dev/xvdcz', ebs: { encrypted: true, volumeSize: 100, volumeType: 'gp2', }, }, ], }, });
and use it:
Vpc vpc; CfnLaunchTemplate myLaunchTemplate; ComputeEnvironment myComputeEnv = ComputeEnvironment.Builder.create(this, "ComputeEnv") .computeResources(ComputeResources.builder() .launchTemplate(LaunchTemplateSpecification.builder() .launchTemplateName((String)myLaunchTemplate.getLaunchTemplateName()) .build()) .vpc(vpc) .build()) .computeEnvironmentName("MyStorageCapableComputeEnvironment") .build();
To import an existing batch compute environment, call ComputeEnvironment.fromComputeEnvironmentArn()
.
Below is an example:
IComputeEnvironment computeEnv = ComputeEnvironment.fromComputeEnvironmentArn(this, "imported-compute-env", "arn:aws:batch:us-east-1:555555555555:compute-environment/My-Compute-Env");
Occasionally, you will need to deviate from the default processing AMI.
ECS Optimized Amazon Linux 2 example:
Vpc vpc; ComputeEnvironment myComputeEnv = ComputeEnvironment.Builder.create(this, "ComputeEnv") .computeResources(ComputeResources.builder() .image(EcsOptimizedAmi.Builder.create() .generation(AmazonLinuxGeneration.AMAZON_LINUX_2) .build()) .vpc(vpc) .build()) .build();
Custom based AMI example:
Vpc vpc; ComputeEnvironment myComputeEnv = ComputeEnvironment.Builder.create(this, "ComputeEnv") .computeResources(ComputeResources.builder() .image(MachineImage.genericLinux(Map.of( "[aws-region]", "[ami-ID]"))) .vpc(vpc) .build()) .build();
Jobs are always submitted to a specific queue. This means that you have to create a queue before you can start submitting jobs. Each queue is mapped to at least one (and no more than three) compute environment. When the job is scheduled for execution, AWS Batch will select the compute environment based on ordinal priority and available capacity in each environment.
ComputeEnvironment computeEnvironment; JobQueue jobQueue = JobQueue.Builder.create(this, "JobQueue") .computeEnvironments(List.of(JobQueueComputeEnvironment.builder() // Defines a collection of compute resources to handle assigned batch jobs .computeEnvironment(computeEnvironment) // Order determines the allocation order for jobs (i.e. Lower means higher preference for job assignment) .order(1) .build())) .build();
Sometimes you might have jobs that are more important than others, and when submitted, should take precedence over the existing jobs. To achieve this, you can create a priority based execution strategy, by assigning each queue its own priority:
ComputeEnvironment sharedComputeEnvs; JobQueue highPrioQueue = JobQueue.Builder.create(this, "JobQueue") .computeEnvironments(List.of(JobQueueComputeEnvironment.builder() .computeEnvironment(sharedComputeEnvs) .order(1) .build())) .priority(2) .build(); JobQueue lowPrioQueue = JobQueue.Builder.create(this, "JobQueue") .computeEnvironments(List.of(JobQueueComputeEnvironment.builder() .computeEnvironment(sharedComputeEnvs) .order(1) .build())) .priority(1) .build();
By making sure to use the same compute environments between both job queues, we will give precedence to the highPrioQueue
for the assigning of jobs to available compute environments.
To import an existing batch job queue, call JobQueue.fromJobQueueArn()
.
Below is an example:
IJobQueue jobQueue = JobQueue.fromJobQueueArn(this, "imported-job-queue", "arn:aws:batch:us-east-1:555555555555:job-queue/High-Prio-Queue");
A Batch Job definition helps AWS Batch understand important details about how to run your application in the scope of a Batch Job. This involves key information like resource requirements, what containers to run, how the compute environment should be prepared, and more. Below is a simple example of how to create a job definition:
import software.amazon.awscdk.services.ecr.*; IRepository repo = Repository.fromRepositoryName(this, "batch-job-repo", "todo-list"); JobDefinition.Builder.create(this, "batch-job-def-from-ecr") .container(JobDefinitionContainer.builder() .image(new EcrImage(repo, "latest")) .build()) .build();
Below is an example of how you can create a Batch Job Definition from a local Docker application.
JobDefinition.Builder.create(this, "batch-job-def-from-local") .container(JobDefinitionContainer.builder() // todo-list is a directory containing a Dockerfile to build the application .image(ContainerImage.fromAsset("../todo-list")) .build()) .build();
You can provide custom log driver and its configuration for the container.
import software.amazon.awscdk.services.ssm.*; JobDefinition.Builder.create(this, "job-def") .container(JobDefinitionContainer.builder() .image(EcrImage.fromRegistry("docker/whalesay")) .logConfiguration(LogConfiguration.builder() .logDriver(LogDriver.AWSLOGS) .options(Map.of("awslogs-region", "us-east-1")) .secretOptions(List.of(ExposedSecret.fromParametersStore("xyz", StringParameter.fromStringParameterName(this, "parameter", "xyz")))) .build()) .build()) .build();
To import an existing batch job definition from its ARN, call JobDefinition.fromJobDefinitionArn()
.
Below is an example:
IJobDefinition job = JobDefinition.fromJobDefinitionArn(this, "imported-job-definition", "arn:aws:batch:us-east-1:555555555555:job-definition/my-job-definition");
To import an existing batch job definition from its name, call JobDefinition.fromJobDefinitionName()
.
If name is specified without a revision then the latest active revision is used.
Below is an example:
// Without revision IJobDefinition job1 = JobDefinition.fromJobDefinitionName(this, "imported-job-definition", "my-job-definition"); // With revision IJobDefinition job2 = JobDefinition.fromJobDefinitionName(this, "imported-job-definition", "my-job-definition:3");