Run AWS Batch workloads with Step Functions

You can integrate Step Functions with AWS Batch to run batch computing workloads in the AWS cloud. This page lists the supported AWS Batch APIs and provides an example Task state to perform a batch-processing task.

To learn about integrating with AWS services in Step Functions, see Integrating services and Passing parameters to a service API in Step Functions.

Key features of Optimized AWS Batch integration

• The Run a Job (.sync) integration pattern is available.

Note that there are no optimizations for the Request Response or Wait for a Callback with Task Token integration patterns.

The following includes a Task state that submits an AWS Batch job and waits for it to complete.

{
  "StartAt": "BATCH_JOB",
  "States": {
    "BATCH_JOB": {
      "Type": "Task",
      "Resource": "arn:aws:states:::batch:submitJob.sync",
      "Parameters": {
        "JobDefinition": "preprocessing",
        "JobName": "PreprocessingBatchJob",
        "JobQueue": "SecondaryQueue",
        "Parameters.$": "$.batchjob.parameters",
        "ContainerOverrides": {
          "ResourceRequirements": [
            {
              "Type": "VCPU",
              "Value": "4"
            }
          ]
        }
      },
      "End": true
    }
  }
}

Supported AWS Batch APIs:

• SubmitJob

  • Request syntax

  • Supported parameters:

    • ArrayProperties

    • ContainerOverrides

    • DependsOn

    • JobDefinition

    • JobName

    • JobQueue

    • Parameters

    • RetryStrategy

    • Timeout

    • Tags

  • Response syntax

Parameters in Step Functions are expressed in PascalCase

Even if the native service API is in camelCase, for example the API action startSyncExecution, you specify parameters in PascalCase, such as: StateMachineArn.

IAM policies for calling AWS Batch

The following example templates show how AWS Step Functions generates IAM policies based on the resources in your state machine definition. For more information, see How Step Functions generates IAM policies for integrated services and Discover service integration patterns in Step Functions.

Because AWS Batch provides partial support for resource-level access control, you must use "Resource": "*".

Run a Job (.sync)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "batch:SubmitJob",
                "batch:DescribeJobs",
                "batch:TerminateJob"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "events:PutTargets",
                "events:PutRule",
                "events:DescribeRule"
            ],
            "Resource": [
               "arn:aws:events:[[region]]:[[accountId]]:rule/StepFunctionsGetEventsForBatchJobsRule"
            ]
        }
    ]
}
        

Request Response

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "batch:SubmitJob"
            ],
            "Resource": "*"
        }
    ]
}