Set the default instance warmup for an Auto Scaling group - Amazon EC2 Auto Scaling

Set the default instance warmup for an Auto Scaling group

CloudWatch collects and aggregates usage data, such as CPU and network I/O, across your Auto Scaling instances. You use these metrics to create scaling policies that adjust the number of instances in your Auto Scaling group as the selected metric's value increases and decreases.

The default instance warmup lets you specify how long after an instance reaches the InService state it waits before contributing usage data to the aggregated metrics. This keeps dynamic scaling from being affected by metrics for individual instances that aren't handling application traffic yet and that might be experiencing temporarily high usage of compute resources.

Default instance warmup is not configured and is not enabled by default. To optimize the performance of your target tracking and step scaling policies, we strongly recommend that you enable the default instance warmup.

If your Auto Scaling group is set to use an instance maintenance policy, or you use an instance refresh to replace instances, you can also prevent instances from being counted toward the minimum healthy percentage before they have finished initializing when you enable the default instance warmup.

Scaling performance considerations

Most applications will benefit from having one default instance warm-up time that applies to all features rather than different warm-up times for different features. For example, if you don't set a default instance warmup, the instance refresh feature uses the health check grace period as the default warm-up time. If you have any target tracking and step scaling policies, they use the value set for the default cooldown as the default warm-up time. If you have any predictive scaling policies, they have no default warm-up time.

While instances are warming up, your dynamic scaling policies only scale out if the metric value from instances that are not warming up is greater than the policy's alarm high threshold (or the target utilization of a target tracking scaling policy). If demand decreases, the intention is to scale in conservatively to protect your application's availability. This blocks scale in for dynamic scaling until the instances finish warming up.

While scaling out, we consider instances that are warming up as part of the theoretical capacity of the group when deciding how many instances to add to the group. Therefore, multiple alarm breaches that fall in the range of the same step adjustment result in a single scaling activity. The intention is to continuously (but not excessively) scale out. For more information, see the Instance warm-up section in the step scaling topic.

If default instance warmup is not enabled, the amount of time an instance waits before sending metrics to CloudWatch and counting it towards the current capacity will vary from instance to instance. In which case, there is the potential for your scaling policies to perform unpredictably compared to the actual workload that is occurring.

For example, consider an application with a recurring on-and-off workload pattern. A predictive scaling policy is used to make recurring decisions about whether to increase the number of instances. Because there is no default warm-up time for predictive scaling policies, the instances start contributing to the aggregated metrics immediately. If these instances have higher resource usage on startup, then adding instances could cause the aggregated metrics to spike. Depending on how long it takes for usage to stabilize, this could impact any dynamic scaling policies that use these metrics. If a dynamic scaling policy's alarm high threshold is breached, then the group increases in size again. While the new instances are warming up, scale-in activities will be blocked.

Choose the default instance warm-up time

The key to setting the default instance warmup is determining how long your instances need to finish initializing and for resource consumption to stabilize after they reach the InService state. When choosing the instance warm-up time, aim for an optimal balance between collecting usage data for legitimate traffic, but minimizing data collection associated with temporary usage spikes on startup.

Suppose you have an Auto Scaling group attached to an Elastic Load Balancing load balancer. When new instances finish launching, they're registered to the load balancer before they enter the InService state. After the instances enter the InService state, resource consumption can still experience temporary spikes and need time to stabilize. For example, resource consumption for an application server that must download and cache large assets takes longer to stabilize than a lightweight web server with no large assets to download. The instance warmup provides the time delay necessary for resource consumption to stabilize.

Important

If you're not sure how much time you need, you could start with 300 seconds and gradually decrease or increase it until you get the best scaling performance for your application. You might need to experiment to get it right. Alternatively, if you have any scaling policies that have their own warm-up time (EstimatedInstanceWarmup), you could use this value to start. For more information, see Find scaling policies with a previously set instance warm-up time.

Consider using lifecycle hooks for use cases where you have configuration tasks or scripts to run on startup. Lifecycle hooks can delay new instances from being put in service until they have finished initializing. They are particularly useful if you have bootstrapping scripts that take a while to complete. If you add a lifecycle hook, you can reduce the value of the default instance warmup. For more information about using lifecycle hooks, see Amazon EC2 Auto Scaling lifecycle hooks.

Enable the default instance warmup for a group

You can enable the default instance warmup when you create an Auto Scaling group. You can also enable it for existing groups.

By enabling the default instance warmup feature, you no longer have to specify values for warm-up parameters for the following features:

Console
To enable the default instance warmup for a new group (console)

When you create the Auto Scaling group, on the Configure advanced options page, under Additional settings, select the Enable default instance warmup option. Choose the warm-up time that you need for your application.

AWS CLI
To enable the default instance warmup for a new group (AWS CLI)

To enable the default instance warmup for an Auto Scaling group, add the --default-instance-warmup option and specify a value, in seconds, from 0 to 3600. After it's enabled, a value of -1 will turn this setting off.

The following create-auto-scaling-group command creates an Auto Scaling group with the name my-asg and enables the default instance warmup with a value of 120 seconds.

aws autoscaling create-auto-scaling-group --auto-scaling-group-name my-asg --default-instance-warmup 120 ...
Tip

If this command throws an error, make sure that you have updated the AWS CLI locally to the latest version.

Console
To enable the default instance warmup for an existing group (console)
  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/, and choose Auto Scaling Groups from the navigation pane.

  2. On the navigation bar at the top of the screen, choose the AWS Region that you created your Auto Scaling group in.

  3. Select the check box next to the Auto Scaling group.

    A split pane opens up in the bottom of the page.

  4. On the Details tab, choose Advanced configurations, Edit.

  5. For Default instance warmup, choose the warm-up time that you need for your application.

  6. Choose Update.

AWS CLI
To enable the default instance warmup for an existing group (AWS CLI)

The following example uses the update-auto-scaling-group command to enable the default instance warmup with a value of 120 seconds for an existing Auto Scaling group named my-asg.

aws autoscaling update-auto-scaling-group --auto-scaling-group-name my-asg --default-instance-warmup 120
Tip

If this command throws an error, make sure that you have updated the AWS CLI locally to the latest version.

Verify the default instance warmup for a group

To verify the default instance warmup for an Auto Scaling group (AWS CLI)

Use the following describe-auto-scaling-groups command. Replace my-asg with the name of your Auto Scaling group.

aws autoscaling describe-auto-scaling-groups --auto-scaling-group-name my-asg

The following is an example response.

{ "AutoScalingGroups": [ { "AutoScalingGroupName": "my-asg", "AutoScalingGroupARN": "arn", ... "DefaultInstanceWarmup": 120 } ] }

Find scaling policies with a previously set instance warm-up time

To identify whether you have policies that have their own warm-up time for EstimatedInstanceWarmup, run the following describe-policies command using the AWS CLI. Replace my-asg with the name of your Auto Scaling group.

aws autoscaling describe-policies --auto-scaling-group-name my-asg --query 'ScalingPolicies[?EstimatedInstanceWarmup!=`null`]'

The following is example output.

[ { "AutoScalingGroupName":"my-asg", "PolicyName":"cpu50-target-tracking-scaling-policy", "PolicyARN":"arn", "PolicyType":"TargetTrackingScaling", "StepAdjustments":[], "EstimatedInstanceWarmup":120, "Alarms":[{ "AlarmARN": "arn:aws:cloudwatch:us-west-2:123456789012:alarm:TargetTracking-my-asg-AlarmHigh-fc0e4183-23ac-497e-9992-691c9980c38e", "AlarmName": "TargetTracking-my-asg-AlarmHigh-fc0e4183-23ac-497e-9992-691c9980c38e" }, { "AlarmARN": "arn:aws:cloudwatch:us-west-2:123456789012:alarm:TargetTracking-my-asg-AlarmLow-61a39305-ed0c-47af-bd9e-471a352ee1a2", "AlarmName": "TargetTracking-my-asg-AlarmLow-61a39305-ed0c-47af-bd9e-471a352ee1a2" }], "TargetTrackingConfiguration":{ "PredefinedMetricSpecification":{ "PredefinedMetricType":"ASGAverageCPUUtilization" }, "TargetValue":50.0, "DisableScaleIn":false }, "Enabled":true }, ... additional policies ... ]

Clear the previously set instance warmup for a scaling policy

After enabling the default instance warmup, update any scaling policies that have still their own warm-up time to clear the previously set value. Otherwise, it will override the default instance warmup.

You can update scaling policies using the console, AWS CLI, or AWS SDKs. This section covers the steps for the console. If you use the AWS CLI or AWS SDKs, make sure you preserve the existing policy configuration, but remove the EstimatedInstanceWarmup property. When you update an existing scaling policy, the policy will be replaced with what you specify when you programmatically call PutScalingPolicy. The original values are not kept.

To clear the previously set instance warmup for a scaling policy (console)
  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/, and choose Auto Scaling Groups from the navigation pane.

  2. Select the check box next to the Auto Scaling group.

    A split pane opens up in the bottom of the page.

  3. On the Automatic scaling tab, in Dynamic scaling policies, choose the policy you're interested in, and then choose Actions, Edit.

  4. For Instance warmup, clear the instance warm-up value to use the default instance warm-up value instead.

  5. Choose Update.