Run AWS Batch workloads with Step Functions
You can integrate Step Functions with AWS Batch to run batch computing workloads in the AWS cloud. This page lists the supported
AWS Batch APIs and provides an example Task
state to perform a batch-processing task.
Step Functions can control certain AWS services directly from Using Amazon States Language to define Step Functions workflows (ASL). To learn more, see Integrating other services and Passing parameters to a service API in Step Functions.
How the Optimized AWS Batch integration is different than the AWS BatchAWS SDK integration
The Run a Job (.sync) integration pattern is available.
Note that there are no optimizations for the Request Response or Wait for a Callback with the Task Token integration patterns.
Supported AWS Batch APIs:
Parameters in Step Functions are expressed in PascalCase
Even if the native service API is in camelCase, for example the API action startSyncExecution
, you specify parameters in PascalCase, such as: StateMachineArn
.
The following includes a Task
state that submits an AWS Batch job and waits
for it to complete.
{
"StartAt": "BATCH_JOB",
"States": {
"BATCH_JOB": {
"Type": "Task",
"Resource": "arn:aws:states:::batch:submitJob.sync",
"Parameters": {
"JobDefinition": "preprocessing",
"JobName": "PreprocessingBatchJob",
"JobQueue": "SecondaryQueue",
"Parameters.$": "$.batchjob.parameters",
"ContainerOverrides": {
"ResourceRequirements": [
{
"Type": "VCPU",
"Value": "4"
}
]
}
},
"End": true
}
}
}
For information about how to configure IAM permissions when using Step Functions with other AWS services, see How Step Functions generates IAM policies for integrated services.