Amazon Elastic Container Service
Developer Guide (API Version 2014-11-13)

Tutorial: Scaling Container Instances with CloudWatch Alarms

The following procedures help you to create an Auto Scaling group for an Amazon ECS cluster. The Auto Scaling group contains container instances that you can scale up (and down) using CloudWatch alarms.

Depending on the Amazon EC2 instance types that you use in your clusters, and quantity of container instances that you have in a cluster, your tasks have a limited amount of resources that they can use while running. Amazon ECS monitors the resources available in the cluster to work with the schedulers to place tasks. If your cluster runs low on any of these resources, such as memory, you are eventually unable to launch more tasks until you add more container instances, reduce the number of desired tasks in a service, or stop some of the running tasks in your cluster to free up the constrained resource.

In this tutorial, you create a CloudWatch alarm using the MemoryReservation metric for your cluster. When the memory reservation of your cluster rises above 75% (meaning that only 25% of the memory in your cluster is available to for new tasks to reserve), the alarm triggers the Auto Scaling group to add another instance and provide more resources for your tasks and services.

Prerequisites

This tutorial assumes that you have enabled CloudWatch metrics for your clusters and services. Metrics are not available until the clusters and services send the metrics to CloudWatch, and you cannot create CloudWatch alarms for metrics that do not exist yet.

Your Amazon ECS container instances require at least version 1.4.0 of the container agent to enable CloudWatch metrics. For information about checking your agent version and updating to the latest version, see Updating the Amazon ECS Container Agent.

Your Amazon ECS container instances also require ecs:StartTelemetrySession permission on the IAM role that you launch your container instances with. If you created your Amazon ECS container instance role before CloudWatch metrics were available for Amazon ECS, then you might need to add this permission. For information about checking your Amazon ECS container instance role and attaching the managed IAM policy for container instances, see To check for the ecsInstanceRole in the IAM console.

Step 1: Create a CloudWatch Alarm for a Metric

After you have enabled CloudWatch metrics for your clusters and services, and the metrics for your cluster are visible in the CloudWatch console, you can set alarms on the metrics. For more information, see Creating Amazon CloudWatch Alarms in the Amazon CloudWatch User Guide.

For this tutorial, you create an alarm on the cluster MemoryReservation metric to alert when the cluster's memory reservation is above 75%.

To create a CloudWatch alarm on a metric

  1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.

  2. On the left navigation, choose Alarms, Create Alarm.

  3. In the CloudWatch Metrics by Category section, choose ECS Metrics > ClusterName.

  4. On the Modify Alarm page, choose the MemoryReservation metric for the default cluster and choose Next.

  5. In the Alarm Threshold section, enter a name and description for your alarm.

    • Name: memory-above-75-pct

    • Description: Cluster memory reservation above 75%

  6. Set the threshold and time period requirement to MemoryReservation greater than 75% for 1 period.

    
                        CloudWatch alarm threshold
  7. (Optional) Configure a notification to send when the alarm is triggered. You can also choose to delete the notification if you don't want to configure one now.

  8. Choose Create Alarm. Now you can use this alarm to trigger your Auto Scaling group to add a container instance when the memory reservation is above 75%.

  9. (Optional) You can also create another alarm that triggers when the memory reservation is below 25%, which you can use to remove a container instance from your Auto Scaling group.

Step 2: Create a Launch Configuration for an Auto Scaling Group

Now that you have enabled CloudWatch metrics and created an alarm based on one of those metrics, you can create a launch configuration and an Auto Scaling group for your cluster. For more information and other configuration options, see the Amazon EC2 Auto Scaling User Guide.

To create an Auto Scaling launch configuration

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

  2. On the left navigation pane, choose Auto Scaling Groups.

  3. On the Welcome to Auto Scaling page, choose Create Auto Scaling Group.

  4. On the Create Auto Scaling Group page, choose Create a new launch configuration.

  5. On the Choose AMI step of the Create Auto Scaling Group wizard, choose Community AMIs.

  6. Choose the ECS-optimized AMI for your Auto Scaling group.

    To use the Amazon ECS-optimized Amazon Linux 2 AMI, type amzn2-ami-ecs in the Search community AMIs field and press the Enter key. Choose Select next to the amzn2-ami-ecs-hvm-2.0.20181112-x86_64-ebs AMI.

    The current Amazon ECS-optimized Amazon Linux 2 AMI IDs by region are listed below for reference.

    Region AMI Name AMI ID EC2 console link
    us-east-2 amzn2-ami-ecs-hvm-2.0.20181112-x86_64-ebs ami-037a92bf1efdb11a2 Launch instance
    us-east-1 amzn2-ami-ecs-hvm-2.0.20181112-x86_64-ebs ami-0a6b7e0cc0b1f464f Launch instance
    us-west-2 amzn2-ami-ecs-hvm-2.0.20181112-x86_64-ebs ami-0c1f4871ebaae6d86 Launch instance
    us-west-1 amzn2-ami-ecs-hvm-2.0.20181112-x86_64-ebs ami-0184f498956de7db5 Launch instance
    eu-west-3 amzn2-ami-ecs-hvm-2.0.20181112-x86_64-ebs ami-0caadc4f0db31a303 Launch instance
    eu-west-2 amzn2-ami-ecs-hvm-2.0.20181112-x86_64-ebs ami-0b5225210a12d9951 Launch instance
    eu-west-1 amzn2-ami-ecs-hvm-2.0.20181112-x86_64-ebs ami-0acc9f8be17a41897 Launch instance
    eu-central-1 amzn2-ami-ecs-hvm-2.0.20181112-x86_64-ebs ami-055aa9664ef169e25 Launch instance
    ap-northeast-2 amzn2-ami-ecs-hvm-2.0.20181112-x86_64-ebs ami-0bdc871079baf9649 Launch instance
    ap-northeast-1 amzn2-ami-ecs-hvm-2.0.20181112-x86_64-ebs ami-0c38293d60d98af86 Launch instance
    ap-southeast-2 amzn2-ami-ecs-hvm-2.0.20181112-x86_64-ebs ami-0eed1c915ea891aca Launch instance
    ap-southeast-1 amzn2-ami-ecs-hvm-2.0.20181112-x86_64-ebs ami-0e28ff4e3f1776d86 Launch instance
    ca-central-1 amzn2-ami-ecs-hvm-2.0.20181112-x86_64-ebs ami-02c80e9173258d289 Launch instance
    ap-south-1 amzn2-ami-ecs-hvm-2.0.20181112-x86_64-ebs ami-0b7c3be99909df6ef Launch instance
    sa-east-1 amzn2-ami-ecs-hvm-2.0.20181112-x86_64-ebs ami-078146697425f25a7 Launch instance
    us-gov-west-1 amzn2-ami-ecs-hvm-2.0.20181112-x86_64-ebs ami-31b5d150 Launch instance
  7. On the Choose Instance Type step of the Create Auto Scaling Group wizard, choose an instance type for your Auto Scaling group and choose Next: Configure details.

  8. On the Configure details step of the Create Auto Scaling Group wizard, enter the following information. The other fields are optional. For more information, see Creating Launch Configurations in the Amazon EC2 Auto Scaling User Guide.

    • Name: Enter a name for your launch configuration.

    • IAM role: Select the ecsInstanceRole for your container instances. If you do not have this role configured, see Amazon ECS Container Instance IAM Role.

    • IP Address Type: Select the IP address type option for your container instances. To allow external traffic to be able to reach your containers, choose Assign a public IP address to every instance.

  9. (Optional) If you have configuration information to pass to your container instances with Amazon EC2 user data, choose Advanced Details and enter your user data in the User data field. For more information, see Amazon ECS Container Agent Configuration.

  10. Choose Next: Add Storage.

  11. On the Add Storage step of the Create Auto Scaling Group wizard, make any storage configuration changes needed for your instances and choose Next: Configure Security Group.

  12. On the Configure Security Group step of the Create Auto Scaling Group wizard, select an existing security group that meets the needs of your containers, or create a new security group, and choose Review.

  13. Review your launch configuration and choose Create launch configuration.

  14. Select a private key to use for connecting to your instances with SSH and choose Create launch configuration. Move on to creating an Auto Scaling group with your new launch configuration.

Step 3: Create an Auto Scaling Group for your Cluster

After the launch configuration is complete, continue with the following procedure to create an Auto Scaling group that uses your launch configuration.

To create an Auto Scaling group

  1. On the Configure Auto Scaling group details step of the Create Auto Scaling Group wizard, enter the following information and then choose Next: Configure scaling policies:

    • Group name: Enter a name for your Auto Scaling group.

    • Group size: Specify the number of container instances with which your Auto Scaling group should start.

    • Network: Select a VPC into which to launch your container instances.

    • Subnet: Select the subnets into which to launch your container instances. For a highly available cluster, we recommend that you enable all of the subnets in the Region.

  2. On the Configure scaling policies step of the Create Auto Scaling Group wizard, choose Use scaling policies to adjust the capacity of this group.

  3. Enter the minimum and maximum number of container instances for your Auto Scaling group.

  4. In the Increase Group Size section, enter the following information:

    • Execute policy when: Select the memory-above-75-pct CloudWatch alarm that you configured earlier.

    • Take the action: Enter the number of instances to add to your cluster when the alarm is triggered.

  5. If you configured an alarm to trigger a group size reduction, set that alarm in the Decrease Group Size section and specify how many instances to remove if that alarm is triggered. Otherwise, collapse the Decrease Group Size section by choosing the X in the upper-right-hand corner of the section.

    Note

    If you configure your Auto Scaling group to remove container instances, any tasks running on the removed container instances are stopped. If your tasks are running as part of a service, Amazon ECS restarts those tasks on another instance if the required resources are available (CPU, memory, ports). However, tasks that were started manually are not restarted automatically.

  6. Choose Review, Create Auto Scaling Group.

Step 4: Verify and Test your Auto Scaling Group

Now that you've created your Auto Scaling group, you should be able to see your instances launching in the Amazon EC2 console Instances page. These instances should register into your Amazon ECS cluster as well after they launch.

To test that your Auto Scaling group is configured properly, create some tasks that consume a considerable amount of memory and start launching them into your cluster. After your cluster exceeds the 75% memory reservation from the CloudWatch alarm for the specified number of periods, you should see a new instance launch in the Amazon EC2 console.

Step 5: Cleaning Up

When you have completed this tutorial, you may choose to keep your Auto Scaling group and Amazon EC2 instances in service for your cluster. However, if you are not actively using these resources, you should consider cleaning them up so your account does not incur unnecessary charges. You can delete your Auto Scaling group to terminate the Amazon EC2 instances within it, but your launch configuration remains intact. You can create a new Auto Scaling group with the launch configuration later, if you choose.

To delete your Auto Scaling group

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

  2. On the left navigation pane, choose Auto Scaling Groups.

  3. Choose the Auto Scaling group that you created earlier.

  4. Choose Actions, Delete.

  5. Choose Yes, Delete.