AWS Batch on AWS Fargate - AWS Batch

AWS Batch on AWS Fargate

AWS Fargate is a technology that you can use with AWS Batch to run containers without having to manage servers or clusters of Amazon EC2 instances. With AWS Fargate, you no longer have to provision, configure, or scale clusters of virtual machines to run containers. This removes the need to choose server types, decide when to scale your clusters, or optimize cluster packing.

When you run your jobs with Fargate resources, you package your application in containers, specify the CPU and memory requirements, define networking and IAM policies, and launch the application. Each Fargate job has its own isolation boundary and does not share the underlying kernel, CPU resources, memory resources, or elastic network interface with another job.

When to use Fargate

For many customers, Fargate is the better choice. Fargate launches and scales the compute to closely match the resource requirements you specify for the container. With Fargate, there is no over-provisioning and paying for additional servers. You don't need to worry about the details of the infrastructure-related parameters like instance type. When the compute environment needs to be scaled up, jobs running on Fargate resources can get started more quickly. It can take a few minutes to spin up a new Amazon EC2 instance, while jobs running on Fargate can be provisioned in about 30 seconds (depending on the container image size, number of jobs, and a few other factors.)

Amazon EC2 is the right choice if jobs require any of these:

  • more than 4 vCPUs

  • more than 30 gibibyte (GiB) of memory

  • a GPU

  • Arm-based AWS Graviton CPU

  • custom AMI

  • any of the linuxParameters parameters

If you have a large number of jobs, Amazon EC2 may be the right choice because jobs can be dispatched at a higher rate to EC2 resources than to Fargate resources, and more jobs can be running concurrently. For more information, see AWS Fargate service quotas in the Amazon Elastic Container Service Developer Guide.

Job definitions on Fargate

AWS Batch jobs on Fargate don't support all of the job definition parameters that are available. Some parameters are not supported at all, and others behave differently for Fargate jobs.

The following list describes job definition parameters that are not valid or otherwise restricted in Fargate jobs.

platformCapabilities

Must be specified as FARGATE.

"platformCapabilities": [ "FARGATE" ]
type

Must be specified as container.

"type": "container"
Parameters in containerProperties
executionRoleArn

Must be specified for jobs running on Fargate resources. For more information, see IAM Roles for Tasks in the Amazon Elastic Container Service Developer Guide.

"executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole"
fargatePlatformConfiguration

(Optional, only for Fargate job definitions). Specifies the Fargate platform version, or LATEST for a recent platform version. Possible values for platformVersion are 1.3.0, 1.4.0, and LATEST (default).

"fargatePlatformConfiguration": { "platformVersion": "1.4.0" }
instanceType
ulimits

Not applicable for jobs running on Fargate resources.

memory
vcpus

These settings must be specified in resourceRequirements

privileged

Either don't specify this parameter, or specify false.

"privileged": false
resourceRequirements

Both memory and vCPU requirements must be specified, using supported values. GPU resources are not supported for jobs running on Fargate resources.

"resourceRequirements": [ {"type": "MEMORY", "value": "512"}, {"type": "VCPU", "value": "0.25"} ]
Parameters in linuxParameters
devices
maxSwap
sharedMemorySize
swappiness
tmpfs

Not applicable for jobs running on Fargate resources.

Parameters in logConfiguration
logDriver

Only awslogs and fluentd are supported. For more information, see Using the awslogs log driver.

Members in networkConfiguration
assignPublicIp

If the private subnet does not have a NAT gateway attached to send traffic to the Internet, assignPublicIp must be "ENABLED". For more information, see For more information, see AWS Batch execution IAM role.

Job queues on Fargate

AWS Batch job queues on Fargate are essentially unchanged. The only restriction is that the compute environments listed in computeEnvironmentOrder must all be Fargate compute environments (FARGATE or FARGATE_SPOT). EC2 and Fargate compute environments can't be mixed.

Compute environments on Fargate

AWS Batch compute environments on Fargate don't support all of the compute environment parameters that are available. Some parameters are not supported at all, and others have specific requirements for Fargate.

The following list describes compute environment parameters that are not valid or otherwise restricted in Fargate jobs.

type

This parameter must be MANAGED.

"type": "MANAGED"
Parameters in the computeResources object
allocationStrategy
bidPercentage
desiredvCpus
imageId
instanceTypes
ec2Configuration
ec2KeyPair
instanceRole
launchTemplate
minvCpus
placementGroup
spotIamFleetRole

These aren't applicable for Fargate compute environments and shouldn't be provided.

subnets

If the subnets listed in this parameter don't have NAT gateways attached, the assignPublicIp parameter in the job definition must be set to ENABLED.

tags

This isn't applicable for Fargate compute environments and shouldn't be provided. To specify tags for Fargate compute environments, use the tags parameter that's not in the computeResources object.

type

This must be either FARGATE or FARGATE_SPOT.

"type": "FARGATE_SPOT"