Creating a compute environment - AWS Batch
To create a managed compute environment using AWS Fargate resourcesTo create a managed compute environment using EC2 resourcesTo create an unmanaged compute environment using EC2 resourcesTo create a managed compute environment using Amazon EKS resources

Creating a compute environment

Before you can run jobs in AWS Batch, you need to create a compute environment. You can create a managed compute environment where AWS Batch manages the Amazon EC2 instances or AWS Fargate resources within the environment based on your specifications. Or, alternatively, you can create an unmanaged compute environment where you handle the Amazon EC2 instance configuration within the environment.

Important

Fargate Spot instances are not supported in the following scenarios:

  • On Amazon Linux containers with ARM64 architecture.

  • Windows containers on AWS Fargate

A job queue will be blocked in these scenarios if a job is submitted to a job queue that only uses Fargate Spot compute environments.

To create a managed compute environment using AWS Fargate resources

  1. Open the AWS Batch console at https://console.aws.amazon.com/batch/.

  2. From the navigation bar, select the AWS Region to use.

  3. In the navigation pane, choose Compute environments.

  4. Choose Create.

  5. Configure the compute environment.

    Note

    Compute environments for Windows containers on AWS Fargate jobs must at least one vCPU.

    1. For Compute environment configuration, choose Fargate.

    2. For Name, specify a unique name for your compute environment. The name can contain up to 128 characters in length. It can contain uppercase and lowercase letters, numbers, hyphens (-), and underscores (_).

    3. For Service role, choose service-linked role that lets the AWS Batch service to make calls to the required AWS API operations on your behalf. For example, choose AWSServiceRoleForBatch. For more information, see Service-linked role permissions for AWS Batch.

    4. (Optional) Expand Tags. To add a tag, choose Add tag. Then, enter a Key name and optional Value. Choose Add tag.

    5. Choose Next page.

  6. In the Instance configuration section:

    1. (Optional) For Use Fargate Spot capacity, turn on Fargate Spot. For information about Fargate Spot, see Using Amazon EC2 Spot and Fargate_SPOT.

    2. For Maximum vCPUs, choose the maximum number of vCPUs that your compute environment can scale out to, regardless of job queue demand.

    3. Choose Next page.

  7. Configure networking.

    Important

    Compute resources need access to communicate with the Amazon ECS service endpoint. This can be through an interface VPC endpoint or through your compute resources having public IP addresses.

    For more information about interface VPC endpoints, see Amazon ECS Interface VPC Endpoints (AWS PrivateLink) in the Amazon Elastic Container Service Developer Guide.

    If you do not have an interface VPC endpoint configured and your compute resources do not have public IP addresses, then they must use network address translation (NAT) to provide this access. For more information, see NAT gateways in the Amazon VPC User Guide. For more information, see Create a VPC.

    1. For Virtual Private Cloud (VPC) ID, choose a VPC where you want to launch your instances.

    2. For Subnets, choose the subnets to use. By default, all subnets within the selected VPC are available.

      Note

      AWS Batch on Fargate doesn't currently support Local Zones. For more information, see Amazon ECS clusters in Local Zones, Wavelength Zones, and AWS Outposts in the Amazon Elastic Container Service Developer Guide.

    3. For Security groups, choose a security group to attach to your instances. By default, the default security group for your VPC is chosen.

    4. Choose Next page.

  8. For Review, review the configuration steps. If you need to make changes, choose Edit. When you're finished, choose Create compute environment.

To create a managed compute environment using EC2 resources

  1. Open the AWS Batch console at https://console.aws.amazon.com/batch/.

  2. From the navigation bar, select the AWS Region to use.

  3. In the navigation pane, choose Compute environments.

  4. Choose Create.

  5. Configure the environment.

    1. For Compute environment configuration, choose Amazon Elastic Compute Cloud (Amazon EC2).

    2. For Orchestration type, choose Managed.

    3. For Name, specify a unique name for your compute environment. The name can contain up to 128 characters in length. It can contain uppercase and lowercase letters, numbers, hyphens (-), and underscores (_).

    4. (Optional) For Service role, choose service-linked role that lets the AWS Batch service make calls to the required AWS API operations on your behalf. For example, choose AWSServiceRoleForBatch. For more information, see Service-linked role permissions for AWS Batch.

    5. For Instance role, choose to create a new instance profile or use an existing instance profile that has the required IAM permissions attached. This instance profile allows the Amazon ECS container instances that are created for your compute environment to make calls to the required AWS API operations on your behalf. For more information, see Amazon ECS instance role. If you choose to create a new instance profile, the required role (ecsInstanceRole) is created for you.

    6. (Optional) Expand Tags.

    7. (Optional) For EC2 tags, choose Add tag to add a tag to resources that are launched in the compute environment. Then, enter a Key name and optional Value. Choose Add tag.

    8. (Optional) For Tags, choose Add tag. Then, enter a Key name and optional Value. Choose Add tag.

      For more information, see Tagging your AWS Batch resources.

    9. Choose Next page.

  6. In the Instance configuration section:

    1. (Optional) For Enable using Spot instances, turn on Spot. For more information, see Spot Instances.

    2. (Spot only) For Maximum % on-demand price, choose the maximum percentage that a Spot Instance price can be when compared with the On-Demand price for that instance type before instances are launched. For example, if your maximum price is 20%, then the Spot price must be less than 20% of the current On-Demand price for that EC2 instance. You always pay the lowest (market) price and never more than your maximum percentage. If you leave this field empty, the default value is 100% of the On-Demand price.

    3. (Spot only) For Spot fleet role, choose an existing Amazon EC2 Spot Fleet IAM role to apply to your Spot compute environment. If you don't already have an existing Amazon EC2 Spot Fleet IAM role, you must create one first. For more information, see Amazon EC2 spot fleet role.

      Important

      To tag your Spot Instances on creation, your Amazon EC2 Spot Fleet IAM role must use the newer AmazonEC2SpotFleetTaggingRole managed policy. The AmazonEC2SpotFleetRole managed policy doesn't have the required permissions to tag Spot Instances. For more information, see Spot Instances not tagged on creation and Tagging your resources.

    4. For Minimum vCPUs, choose the minimum number of vCPUs that your compute environment maintains, regardless of job queue demand.

    5. For Desired vCPUs, choose the number of vCPUs that your compute environment launches with. As your job queue demand increases, AWS Batch can increase the desired number of vCPUs in your compute environment and add EC2 instances, up to the maximum vCPUs. As demand decreases, AWS Batch can decrease the desired number of vCPUs in your compute environment and remove instances, down to the minimum vCPUs.

    6. For Maximum vCPUs, choose the maximum number of vCPUs that your compute environment can scale out to, regardless of job queue demand.

    7. For Allowed instance types, choose the Amazon EC2 instance types that can be launched. You can specify instance families to launch any instance type within those families (for example, c5, c5n, or p3). Or, you can specify specific sizes within a family (such as c5.8xlarge). Metal instance types aren't in the instance families. For example, c5 doesn't include c5.metal. You can also choose optimal to select instance types (from the C4, M4, and R4 instance families) that match the demand of your job queues.

      Note

      When you create a compute environment, the instance types that you select for the compute environment must share the same architecture. For example, you can't mix x86 and ARM instances in the same compute environment.

      Note

      AWS Batch will scale GPUs based on the required amount in your job queues. To use GPU scheduling, the compute environment must include instance types from the p2, p3, p4, p5, g3, g3s, g4, or g5 families.

      Note

      Currently, optimal uses instance types from the C4, M4, and R4 instance families. In AWS Regions that don't have instance types from those instance families, instance types from the C5, M5, and R5 instance families are used.

    8. Expand Additional configuration.

    9. (Optional) For Placement group, enter a placement group name to group resources in the compute environment.

    10. (Optional) For EC2 key pair, choose a public and private key pair as security credentials when you connect to the instance. For more information about Amazon EC2 key pairs, see Amazon EC2 key pairs and Linux instances.

    11. For Allocation strategy, choose the allocation strategy to use when selecting instance types from the list of allowed instance types. BEST_FIT_PROGRESSIVE is usually the better choice for EC2 On-Demand compute environments, SPOT_CAPACITY_OPTIMIZED, and SPOT_PRICE_CAPACITY_OPTIMIZED for EC2 Spot compute environments. For more information, see Allocation strategies.

    12. (Optional) For EC2 configuration choose Image type and Image ID override values to provide information for AWS Batch to select Amazon Machine Images (AMIs) for instances in the compute environment. If the Image ID override isn't specified for each Image type, AWS Batch selects a recent Amazon ECS optimized AMI. If no Image type is specified, the default is a Amazon Linux 2 for non-GPU, non AWS Graviton instance.

      Important

      To use a custom AMI, choose the image type and then enter the custom AMI ID in the Image ID override box.

      Amazon Linux 2

      Default for all AWS Graviton-based instance families (for example, C6g, M6g, R6g, and T4g) and can be used for all non-GPU instance types.

      Amazon Linux 2 (GPU)

      Default for all GPU instance families (for example P4 and G4) and can be used for all non AWS Graviton-based instance types.

      Amazon Linux

      Can be used for non-GPU, non AWS Graviton instance families. The standard support for Amazon Linux AMI has ended. For more information, see Amazon Linux AMI.

      Note

      The AMI that you choose for a compute environment must match the architecture of the instance types that you want to use for that compute environment. For example, if your compute environment uses A1 instance types, the compute resource AMI that you choose must support Arm instances. Amazon ECS vends both x86 and Arm versions of the Amazon ECS optimized Amazon Linux 2 AMI. For more information, see Amazon ECS optimized Amazon Linux 2 AMI in the Amazon Elastic Container Service Developer Guide.

    13. (Optional) For Launch template, select an existing Amazon EC2 launch template to configure your compute resources. The default version of the template is automatically populated. For more information, see Launch template support.

      Note

      In a launch template, you can specify a custom AMI that you created.

    14. (Optional) For Launch template version, enter $Default, $Latest, or a specific version number to use.

      Important

      If the version parameter of the launch template is $Default or $Latest, the default or latest version of the specified launch template is evaluated during an infrastructure update. If a different AMI ID is selected by the default or the latest version of the launch template is selected, that AMI ID is used in the update. For more information, see Updating the AMI ID.

    15. Choose Next page.

  7. In the Network configuration section:

    Important

    Compute resources need access to communicate with the Amazon ECS service endpoint. This can be through an interface VPC endpoint or through your compute resources having public IP addresses.

    For more information about interface VPC endpoints, see Amazon ECS Interface VPC Endpoints (AWS PrivateLink) in the Amazon Elastic Container Service Developer Guide.

    If you do not have an interface VPC endpoint configured and your compute resources do not have public IP addresses, then they must use network address translation (NAT) to provide this access. For more information, see NAT gateways in the Amazon VPC User Guide. For more information, see Create a VPC.

    1. For Virtual Private Cloud (VPC) ID, choose a VPC where to launch your instances.

    2. For Subnets, choose the subnets to use. By default, all subnets within the selected VPC are available.

      Note

      AWS Batch on Amazon EC2 supports Local Zones. For more information, see Local Zones in the Amazon EC2 User Guide for Linux Instances and Amazon ECS clusters in Local Zones, Wavelength Zones, and AWS Outposts in the Amazon Elastic Container Service Developer Guide.

    3. (Optional) For Security groups, choose a security group to attach to your instances. By default, the default security group for your VPC is chosen.

  8. Choose Next page.

  9. For Review, review the configuration steps. If you need to make changes, choose Edit. When you're finished, choose Create compute environment.

To create an unmanaged compute environment using EC2 resources

  1. Open the AWS Batch console at https://console.aws.amazon.com/batch/.

  2. From the navigation bar, select the AWS Region to use.

  3. On the Compute Environments page, choose Create.

  4. Configure the environment.

    1. For Compute environment configuration, choose Amazon Elastic Compute Cloud (Amazon EC2).

    2. For Orchestration type, choose Unmanaged.

  5. For Name, specify a unique name for your compute environment. The name can be up to 128 characters in length. It can contain uppercase and lowercase letters, numbers, hyphens (-), and underscores (_).

  6. (Optional) For Service role, choose a role that lets the AWS Batch service to make calls to the required AWS API operations on your behalf. For example, choose AWSBatchServiceRole. For more information, see Using service-linked roles for AWS Batch..

  7. For Maximum vCPUs, choose the maximum number of vCPUs that your compute environment can scale out to, regardless of job queue demand.

  8. (Optional) Expand Tags. To add a tag, choose Add tag. Then, enter a Key name and optional Value. Choose Add tag. For more information, see Tagging your AWS Batch resources.

  9. Choose Next page.

  10. For Review, review the configuration steps. If you need to make changes, choose Edit. When you're finished, choose Create compute environment.

To create a managed compute environment using Amazon EKS resources

  1. Open the AWS Batch console at https://console.aws.amazon.com/batch/.

  2. From the navigation bar, select the AWS Region to use.

  3. In the navigation pane, choose Compute environments.

  4. Choose Create.

  5. For Compute environment configuration, choose Amazon Elastic Kubernetes Service (Amazon EKS).

  6. For Name, specify a unique name for your compute environment. The name can be up to 128 characters in length. It can contain uppercase and lowercase letters, numbers, hyphens (-), and underscores (_).

  7. For Instance role, choose an existing instance profile that has the required IAM permissions attached.

    Note

    To create a compute environment in the AWS Batch console, choose an instance profile that has the eks:ListClusters and eks:DescribeCluster permissions.

  8. For EKS cluster, choose an existing Amazon EKS cluster.

  9. For Namespace, enter a Kubernetes namespace to group your AWS Batch processes in the cluster.

  10. (Optional) Expand Tags. Choose Add tag and then enter a key-value pair.

  11. Choose Next page.

  12. (Optional) For Use EC2 Spot Instances, turn on Enable using Spot instances to use Amazon EC2 Spot Instances.

  13. (Spot only) For Maximum % on-demand price, choose the maximum percentage that a Spot Instance price can be when compared with the On-Demand price for that instance type before instances are launched. For example, if your maximum price is 20%, then the Spot price must be less than 20% of the current On-Demand price for that EC2 instance. You always pay the lowest (market) price and never more than your maximum percentage. If you leave this field empty, the default value is 100% of the On-Demand price.

  14. (Spot only) For Spot fleet role, choose the Amazon EC2 Spot fleet IAM role for the SPOT compute environment.

    Important

    This role is required if the allocation strategy is set to BEST_FIT or not specified.

  15. (Optional) For Minimum vCPUs, choose the minimum number of vCPUs that your compute environment maintains, regardless of job queue demand.

  16. (Optional) For Maximum vCPUs, choose the maximum number of vCPUs that your compute environment can scale out to, regardless of job queue demand.

  17. For Allowed instance types, choose the Amazon EC2 instance types that can be launched. You can specify instance families to launch any instance type within those families (for example, c5, c5n, or p3). Or, you can specify specific sizes within a family (for example, c5.8xlarge). Metal instance types aren't in the instance families. For example, c5 doesn't include c5.metal. You can also choose optimal to select instance types (from the C4, M4, and R4 instance families) because you need that match the demand of your job queues.

    Note

    When you create a compute environment, the instance types that you select for the compute environment must share the same architecture. For example, you can't mix x86 and ARM instances in the same compute environment.

    Note

    AWS Batch scales GPUs based on the required amount in your job queues. To use GPU scheduling, the compute environment must include instance types from the p2, p3, p4, p5, g3, g3s, g4, or g5 families.

    Note

    Currently, optimal uses instance types from the C4, M4, and R4 instance families. In AWS Regions that don't have instance types from those instance families, instance types from the C5, M5, and R5 instance families are used.

  18. (Optional) Expand Additional configuration.

    1. (Optional) For Placement group, enter a placement group name to group resources in the compute environment.

    2. For Allocation strategy, choose BEST_FIT_PROGRESSIVE.

    3. (Optional) For Amazon Machine Images (AMIs) Configuration, choose Add amazon machine images (amis) configuration. Then, choose an Image Type, enter an Image ID override, and Kubernetes version.

      Important

      To use a custom AMI, choose the image type and then enter the custom AMI ID in the Image ID override box.

      Note

      If the Image ID override isn't specified for each Image type, AWS Batch selects a recent Amazon ECS optimized AMI. If no Image type is specified, the default is a Amazon Linux 2 for non-GPU, non AWS Graviton instance.

      Amazon Linux 2

      Default for all AWS Graviton-based instance families (for example, C6g, M6g, R6g, and T4g) and can be used for all non-GPU instance types.

      Amazon Linux 2 (GPU)

      Default for all GPU instance families (for example, P4 and G4) and can be used for all non AWS Graviton-based instance types.

    4. (Optional) For Launch template, choose an existing launch template.

    5. (Optional) For Launch template version, enter $Default, $Latest, or a version number.

  19. Choose Next page.

  20. For Virtual Private Cloud (VPC) ID, choose a VPC where to launch the instances.

  21. For Subnets, choose the subnets to use. By default, all subnets within the selected VPC are available.

    Note

    AWS Batch on Amazon EKS supports Local Zones. For more information, see Amazon EKS and AWS Local Zones in the Amazon EKS User Guide.

  22. (Optional) For Security groups, choose a security group to attach to your instances. By default, the default security group for your VPC is selected.

  23. Choose Next page.

  24. For Review, review the configuration steps. If you need to make changes, choose Edit. When you're finished, choose Create compute environment.