Amazon Elastic Container Service
Developer Guide (API Version 2014-11-13)

Tutorial: Scaling Container Instances with CloudWatch Alarms

The following procedures help you to create an Auto Scaling group for an Amazon ECS cluster. The Auto Scaling group contains container instances that you can scale up (and down) using CloudWatch alarms.

Depending on the Amazon EC2 instance types that you use in your clusters, and quantity of container instances that you have in a cluster, your tasks have a limited amount of resources that they can use while running. Amazon ECS monitors the resources available in the cluster to work with the schedulers to place tasks. If your cluster runs low on any of these resources, such as memory, you are eventually unable to launch more tasks until you add more container instances, reduce the number of desired tasks in a service, or stop some of the running tasks in your cluster to free up the constrained resource.

In this tutorial, you create a CloudWatch alarm using the MemoryReservation metric for your cluster. When the memory reservation of your cluster rises above 75% (meaning that only 25% of the memory in your cluster is available to for new tasks to reserve), the alarm triggers the Auto Scaling group to add another instance and provide more resources for your tasks and services.

Prerequisites

This tutorial assumes that you have enabled CloudWatch metrics for your clusters and services. Metrics are not available until the clusters and services send the metrics to CloudWatch, and you cannot create CloudWatch alarms for metrics that do not exist yet.

Your Amazon ECS container instances require at least version 1.4.0 of the container agent to enable CloudWatch metrics. For information about checking your agent version and updating to the latest version, see Updating the Amazon ECS Container Agent.

Your Amazon ECS container instances also require ecs:StartTelemetrySession permission on the IAM role that you launch your container instances with. If you created your Amazon ECS container instance role before CloudWatch metrics were available for Amazon ECS, then you might need to add this permission. For information about checking your Amazon ECS container instance role and attaching the managed IAM policy for container instances, see To check for the ecsInstanceRole in the IAM console.

Step 1: Create a CloudWatch Alarm for a Metric

After you have enabled CloudWatch metrics for your clusters and services, and the metrics for your cluster are visible in the CloudWatch console, you can set alarms on the metrics. For more information, see Creating Amazon CloudWatch Alarms in the Amazon CloudWatch User Guide.

For this tutorial, you create an alarm on the cluster MemoryReservation metric to alert when the cluster's memory reservation is above 75%.

To create a CloudWatch alarm on a metric

  1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.

  2. On the left navigation, choose Alarms, Create Alarm.

  3. In the CloudWatch Metrics by Category section, choose ECS Metrics > ClusterName.

  4. On the Modify Alarm page, choose the MemoryReservation metric for the default cluster and choose Next.

  5. In the Alarm Threshold section, enter a name and description for your alarm.

    • Name: memory-above-75-pct

    • Description: Cluster memory reservation above 75%

  6. Set the threshold and time period requirement to MemoryReservation greater than 75% for 1 period.

    
                        CloudWatch alarm threshold
  7. (Optional) Configure a notification to send when the alarm is triggered. You can also choose to delete the notification if you don't want to configure one now.

  8. Choose Create Alarm. Now you can use this alarm to trigger your Auto Scaling group to add a container instance when the memory reservation is above 75%.

  9. (Optional) You can also create another alarm that triggers when the memory reservation is below 25%, which you can use to remove a container instance from your Auto Scaling group.

Step 2: Create a Launch Configuration for an Auto Scaling Group

Now that you have enabled CloudWatch metrics and created an alarm based on one of those metrics, you can create a launch configuration and an Auto Scaling group for your cluster. For more information and other configuration options, see the Amazon EC2 Auto Scaling User Guide.

To create an Auto Scaling launch configuration

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

  2. On the left navigation pane, choose Auto Scaling Groups.

  3. On the Welcome to Auto Scaling page, choose Create Auto Scaling Group.

  4. On the Create Auto Scaling Group page, choose Create a new launch configuration.

  5. On the Choose AMI step of the Create Auto Scaling Group wizard, choose Community AMIs.

  6. Choose the ECS-optimized AMI for your Auto Scaling group.

    To use the Amazon ECS-optimized Amazon Linux 2 AMI, type amzn2-ami-ecs in the Search community AMIs field and press the Enter key. Choose Select next to the amzn2-ami-ecs-hvm-2.0.20190510-x86_64-ebs AMI.

    The following table lists the current Amazon ECS-optimized Amazon Linux 2 AMI IDs by Region.

    Region AMI Name AMI ID EC2 Console Link
    us-east-2 amzn2-ami-ecs-hvm-2.0.20190510-x86_64-ebs ami-012ca23958772cf72 Launch instance
    us-east-1 amzn2-ami-ecs-hvm-2.0.20190510-x86_64-ebs ami-00cf4737e238866a3 Launch instance
    us-west-1 amzn2-ami-ecs-hvm-2.0.20190510-x86_64-ebs ami-06d87f0156b1d4407 Launch instance
    us-west-2 amzn2-ami-ecs-hvm-2.0.20190510-x86_64-ebs ami-0a9f5be2a016dccad Launch instance
    ap-east-1 amzn2-ami-ecs-hvm-2.0.20190510-x86_64-ebs ami-067f4f7124e746edd Launch instance
    ap-northeast-1 amzn2-ami-ecs-hvm-2.0.20190510-x86_64-ebs ami-0e52aad6ac7733a6a Launch instance
    ap-northeast-2 amzn2-ami-ecs-hvm-2.0.20190510-x86_64-ebs ami-08834c8c57e502d6d Launch instance
    ap-south-1 amzn2-ami-ecs-hvm-2.0.20190510-x86_64-ebs ami-04322e867758d97a8 Launch instance
    ap-southeast-1 amzn2-ami-ecs-hvm-2.0.20190510-x86_64-ebs ami-0047bfdb16f1f6781 Launch instance
    ap-southeast-2 amzn2-ami-ecs-hvm-2.0.20190510-x86_64-ebs ami-09475847322e5566f Launch instance
    ca-central-1 amzn2-ami-ecs-hvm-2.0.20190510-x86_64-ebs ami-0498c464ec4d2ba83 Launch instance
    eu-central-1 amzn2-ami-ecs-hvm-2.0.20190510-x86_64-ebs ami-096a38c97b80cd8ec Launch instance
    eu-north-1 amzn2-ami-ecs-hvm-2.0.20190510-x86_64-ebs ami-0dddc4daca44e6e99 Launch instance
    eu-west-1 amzn2-ami-ecs-hvm-2.0.20190510-x86_64-ebs ami-0c5abd45f676aab4f Launch instance
    eu-west-2 amzn2-ami-ecs-hvm-2.0.20190510-x86_64-ebs ami-0204aa6a92a54561e Launch instance
    eu-west-3 amzn2-ami-ecs-hvm-2.0.20190510-x86_64-ebs ami-07273195833e4f20c Launch instance
    sa-east-1 amzn2-ami-ecs-hvm-2.0.20190510-x86_64-ebs ami-00d851648873aaabc Launch instance
    us-gov-east-1 amzn2-ami-ecs-hvm-2.0.20190510-x86_64-ebs ami-06473be43b0f77600 Launch instance
    us-gov-west-1 amzn2-ami-ecs-hvm-2.0.20190510-x86_64-ebs ami-607c0001 Launch instance
  7. On the Choose Instance Type step of the Create Auto Scaling Group wizard, choose an instance type for your Auto Scaling group and choose Next: Configure details.

  8. On the Configure details step of the Create Auto Scaling Group wizard, enter the following information. The other fields are optional. For more information, see Creating Launch Configurations in the Amazon EC2 Auto Scaling User Guide.

    • Name: Enter a name for your launch configuration.

    • IAM role: Select the ecsInstanceRole for your container instances. If you do not have this role configured, see Amazon ECS Container Instance IAM Role.

    • IP Address Type: Select the IP address type option for your container instances. To allow external traffic to be able to reach your containers, choose Assign a public IP address to every instance.

  9. (Optional) If you have configuration information to pass to your container instances with Amazon EC2 user data, choose Advanced Details and enter your user data in the User data field. For more information, see Amazon ECS Container Agent Configuration.

  10. Choose Next: Add Storage.

  11. On the Add Storage step of the Create Auto Scaling Group wizard, make any storage configuration changes needed for your instances and choose Next: Configure Security Group.

  12. On the Configure Security Group step of the Create Auto Scaling Group wizard, select an existing security group that meets the needs of your containers, or create a new security group, and choose Review.

  13. Review your launch configuration and choose Create launch configuration.

  14. Select a private key to use for connecting to your instances with SSH and choose Create launch configuration. Move on to creating an Auto Scaling group with your new launch configuration.

Step 3: Create an Auto Scaling Group for your Cluster

After the launch configuration is complete, continue with the following procedure to create an Auto Scaling group that uses your launch configuration.

To create an Auto Scaling group

  1. On the Configure Auto Scaling group details step of the Create Auto Scaling Group wizard, enter the following information and then choose Next: Configure scaling policies:

    • Group name: Enter a name for your Auto Scaling group.

    • Group size: Specify the number of container instances with which your Auto Scaling group should start.

    • Network: Select a VPC into which to launch your container instances.

    • Subnet: Select the subnets into which to launch your container instances. For a highly available cluster, we recommend that you enable all of the subnets in the Region.

  2. On the Configure scaling policies step of the Create Auto Scaling Group wizard, choose Use scaling policies to adjust the capacity of this group.

  3. Enter the minimum and maximum number of container instances for your Auto Scaling group.

  4. In the Increase Group Size section, enter the following information:

    • Execute policy when: Select the memory-above-75-pct CloudWatch alarm that you configured earlier.

    • Take the action: Enter the number of instances to add to your cluster when the alarm is triggered.

  5. If you configured an alarm to trigger a group size reduction, set that alarm in the Decrease Group Size section and specify how many instances to remove if that alarm is triggered. Otherwise, collapse the Decrease Group Size section by choosing the X in the upper-right-hand corner of the section.

    Note

    If you configure your Auto Scaling group to remove container instances, any tasks running on the removed container instances are stopped. If your tasks are running as part of a service, Amazon ECS restarts those tasks on another instance if the required resources are available (CPU, memory, ports). However, tasks that were started manually are not restarted automatically.

  6. Choose Review, Create Auto Scaling Group.

Step 4: Verify and Test your Auto Scaling Group

Now that you've created your Auto Scaling group, you should be able to see your instances launching in the Amazon EC2 console Instances page. These instances should register into your Amazon ECS cluster as well after they launch.

To test that your Auto Scaling group is configured properly, create some tasks that consume a considerable amount of memory and start launching them into your cluster. After your cluster exceeds the 75% memory reservation from the CloudWatch alarm for the specified number of periods, you should see a new instance launch in the Amazon EC2 console.

Step 5: Cleaning Up

When you have completed this tutorial, you may choose to keep your Auto Scaling group and Amazon EC2 instances in service for your cluster. However, if you are not actively using these resources, you should consider cleaning them up so your account does not incur unnecessary charges. You can delete your Auto Scaling group to terminate the Amazon EC2 instances within it, but your launch configuration remains intact. You can create a new Auto Scaling group with the launch configuration later, if you choose.

To delete your Auto Scaling group

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

  2. On the left navigation pane, choose Auto Scaling Groups.

  3. Choose the Auto Scaling group that you created earlier.

  4. Choose Actions, Delete.

  5. Choose Yes, Delete.