AWS Batch
User Guide

Creating a Compute Resource AMI

You can create your own custom compute resource AMI to use for your managed and unmanaged compute environments, provided that you follow the Compute Resource AMI Specification. After you have created your custom AMI, you can create a compute environment that uses that AMI, associate it with a job queue, and then start submitting jobs to that queue.

To create a custom compute resource AMI

  1. Choose a base AMI to start from. The base AMI must use HVM virtualization, and it cannot be a Windows AMI.

    The Amazon ECS-optimized AMI is the default AMI for compute resources in managed compute environments. The Amazon ECS-optimized AMI is preconfigured and tested on AWS Batch by AWS engineers. It is the simplest AMI for you to get started and to get your compute resources running on AWS quickly. For more information, see Amazon ECS-Optimized AMI in the Amazon Elastic Container Service Developer Guide.

    Alternatively, you can choose another Amazon Linux variant and install the ecs-init package with the following command:

    sudo yum install -y ecs-init

    For example, if you want to run GPU workloads on your AWS Batch compute resources, you could start with the Amazon Linux Deep Learning AMI and configure it to be able to run AWS Batch jobs. For more information, see Creating a GPU Workload AMI.

    Important

    If you choose a base AMI that does not support the ecs-init package, you must configure a way to start the Amazon ECS agent at boot and keep it running. To view several example user data configuration scripts that use systemd to start and monitor the Amazon ECS container agent, see Example Container Instance User Data Configuration Scripts in the Amazon Elastic Container Service Developer Guide.

  2. Launch an instance from your selected base AMI with the appropriate storage options for your AMI. You can configure the size and number of attached Amazon EBS volumes, or instance storage volumes if the instance type you've selected supports them. For more information, see Launching an Instance and Amazon EC2 Instance Store in the Amazon EC2 User Guide for Linux Instances.

  3. Connect to your instance with SSH and perform any necessary configuration tasks, such as:

    • Install the Amazon ECS container agent. For more information, see Installing the Amazon ECS Container Agent in the Amazon Elastic Container Service Developer Guide.

    • Configuring a script to format instance store volumes.

    • Adding instance store volume or Amazon EFS file systems to the /etc/fstab file so that they are mounted at boot.

    • Configuring Docker options (enable debugging, adjust base image size, and so on).

    • Installing packages or copying files.

    For more information, see Connecting to Your Linux Instance Using SSH in the Amazon EC2 User Guide for Linux Instances.

  4. If you started the Amazon ECS container agent on your instance, you must stop it and remove the persistent data checkpoint file before creating your AMI; otherwise, the agent will not start on instances that are launched from your AMI.

    1. Stop the Amazon ECS container agent.

      sudo stop ecs
    2. Remove the persistent data checkpoint file. By default, this file is located at /var/lib/ecs/data/ecs_agent_data.json. Use the following command to remove the file.

      sudo rm -rf /var/lib/ecs/data/ecs_agent_data.json
  5. Create a new AMI from your running instance. For more information, see Creating an Amazon EBS-Backed Linux AMI in the Amazon EC2 User Guide for Linux Instances guide.

To use your new AMI with AWS Batch

  1. When the AMI creation process is complete, create a compute environment with your new AMI (be sure to select Enable user-specified AMI ID and specify your custom AMI ID in Step 7). For more information, see Creating a Compute Environment.

  2. Create a job queue and associate your new compute environment. For more information, see Creating a Job Queue.

  3. (Optional) Submit a sample job to your new job queue. For more information, see Example Job Definitions, Creating a Job Definition, and Submitting a Job.