Container instance draining - Amazon Elastic Container Service

Container instance draining

There might be times when you need to remove a container instance from your cluster; for example, to perform system updates, update the Docker daemon, or to scale down the cluster capacity. Amazon ECS provides the ability to transition a container instance to a DRAINING status. This is referred to as container instance draining. When a container instance is set to DRAINING, Amazon ECS prevents new tasks from being scheduled for placement on the container instance. Any tasks that are part of a service that are in a PENDING state are stopped immediately. If there is available container instance capacity in the cluster, the service scheduler will start replacement tasks. If there isn't enough container instance capacity, a service event message will be sent indicating the issue.

Tasks that are part of a service on the container instance that are in a RUNNING state are transitioned to a STOPPED state. The service scheduler attempts to replace the tasks according to the service's deployment configuration parameters, minimumHealthyPercent and maximumPercent. For more information, see Service definition parameters.

  • If minimumHealthyPercent is below 100%, the scheduler can ignore desiredCount temporarily during task replacement. For example, desiredCount is four tasks, a minimum of 50% allows the scheduler to stop two existing tasks before starting two new tasks. If the minimum is 100%, the service scheduler can't remove existing tasks until the replacement tasks are considered healthy. If tasks for services that do not use a load balancer are in the RUNNING state, they are considered healthy. Tasks for services that use a load balancer are considered healthy if they are in the RUNNING state and the container instance they are hosted on is reported as healthy by the load balancer.

  • The maximumPercent parameter represents an upper limit on the number of running tasks during task replacement, which enables you to define the replacement batch size. For example, if desiredCount of four tasks, a maximum of 200% starts four new tasks before stopping the four tasks to be drained (provided that the cluster resources required to do this are available). If the maximum is 100%, then replacement tasks can't start until the draining tasks have stopped.

Any standalone tasks in the PENDING or RUNNING state are unaffected; you must wait for them to stop on their own or stop them manually.

A container instance has completed draining when all tasks running on the instance transition to a STOPPED state. The container instance remains in a DRAINING state until it is activated again or deleted. You can verify the state of the tasks on the container instance by using the ListTasks operation with the containerInstance parameter to get a list of tasks on the instance followed by a DescribeTasks operation with the Amazon Resource Name (ARN) or ID of each task to verify the task state.

When you are ready for the container instance to start hosting tasks again, you change the state of the container instance from DRAINING to ACTIVE. The Amazon ECS service scheduler will then consider the container instance for task placement again.

Draining container instances

You can use the UpdateContainerInstancesState API action or the update-container-instances-state command to change the status of a container instance to DRAINING.

The following steps can be used to set a container instance to draining using the AWS Management Console.

To set your container instance to DRAINING (AWS Management Console)

  1. Open the Amazon ECS console at https://console.aws.amazon.com/ecs/.

  2. In the navigation pane, choose Clusters and select the cluster.

  3. Choose the ECS Instances tab and select the check box for each container instance you want to drain.

  4. Choose Actions, Drain instances.

  5. After the instances are processed, choose Done.

  6. When you want to activate the container instances again, repeat these same steps but choose Activate instances from the Actions menu.