Menu
Amazon EMR
Management Guide

Using Automatic Scaling in Amazon EMR

Automatic scaling in Amazon EMR allows you to programmatically scale out and scale in core nodes and task nodes in a cluster based on rules that you specify in a scaling policy. The scaling policy is part of an instance group configuration. You can specify a policy during initial configuration of an instance group, or by modifying an instance group in an existing cluster, even when that instance group is active. Each instance group in a cluster, except the master instance group, can have its own scaling policy, which consists of scale-out and scale-in rules. Scale-out and scale-in rules can be configured independently, with different parameters for each rule.

You can configure scaling policies using the AWS Management Console, the AWS CLI, or the Amazon EMR API. When you use the AWS CLI or Amazon EMR API, you specify the scaling policy in JSON format. When you initially create a scaling policy using the console, a default policy suitable for many applications is pre-configured to help you get started. You can delete or modify the default rules.

Even though automatic scaling allows you to adjust EMR cluster capacity on-the-fly, you should still consider baseline workload requirements and plan your node and instance group configurations. For more information, see Cluster Configuration Guidelines.

Note

For most workloads, setting up both scale-in and scale-out rules is desirable to optimize resource utilization. Setting either rule without the other means that you need to manually resize the instance count after a scaling activity. In other words, this sets up a "one-way" automatic scale-out or scale-in policy with a manual reset.

Creating the IAM Role for Automatic Scaling

Automatic scaling in Amazon EMR requires an IAM role with permissions to add and terminate instances when scaling activities are triggered. A default role, EMR_AutoScaling_DefaultRole configured with the appropriate role policy and trust policy, is available for this purpose. Automatic scaling uses this IAM role to scale nodes on your behalf.

When you create a new cluster, you must create this role. Automatic scaling is not available on clusters that do not have this role, and the role cannot be added after the cluster is created. When you create a scaling policy using the AWS Management Console for Amazon EMR, the role is created by default. When you create a cluster that has an automatic scaling policy using the AWS CLI, you must use the --auto-scaling-role EMR_AutoScaling_DefaultRole command to create the role. For more information, see Creating a Cluster with an Automatic Scaling Policy Applied to an Instance Group.

Understanding Automatic Scaling Rules

When a scale-out rule triggers a scaling activity for an instance group, Amazon EC2 instances are added to the instance group according to your rules. New nodes can be used by applications such as Apache Spark and Apache Hive as soon as the Amazon EC2 instance enters the InService state. You can also set up a scale-in rule that terminates instances and removes nodes. For more information about the lifecycle of Amazon EC2 instances that scale automatically, see Auto Scaling Lifecycle in the Auto Scaling User Guide.

You can configure how a cluster terminates Amazon EC2 instances. You can choose to either terminate at the Amazon EC2 instance-hour boundary for billing, or upon task completion. This setting applies both to automatic scaling and to manual resizing operations. For more information about this configuration see Cluster Scale-Down.

The following parameters for each rule in a policy determine automatic scaling behavior.

Note

The parameters listed here are based on the AWS Management Console for Amazon EMR. When you use the AWS CLI or Amazon EMR API, additional advanced configuration options are available. For more information about advanced options, see SimpleScalingPolicyConfiguration in the Amazon EMR API Reference.

  • Maximum instances and minimum instances. The Maximum instances constraint specifies the maximum number of Amazon EC2 instances that can be in the instance group, and applies to all scale-out rules. Similarly, the Minimum instances constraint specifies the minimum number of Amazon EC2 instances and applies to all scale-in rules.

  • The Rule name, which must be unique within the policy.

  • The scaling adjustment, which determines the number of EC2 instances to add (for scale-out rules) or terminate (for scale-in rules) during the scaling activity triggered by the rule.

  • The CloudWatch metric, which is watched for an alarm condition.

  • A comparison operator, which is used to compare the CloudWatch metric to the Threshold value and determine a trigger condition.

  • An evaluation period, in five-minute increments, for which the CloudWatch metric must be in a trigger condition before scaling activity is triggered.

  • A Cooldown period, which determines the amount of time that must elapse between a scaling activity started by a rule and the start of the next scaling activity, regardless of the rule that triggers it. When an instance group has finished a scaling activity and reached its post-scale state, the cooldown period provides an opportunity for the CloudWatch metrics that might trigger subsequent scaling activities to stabilize. For more information, see Auto Scaling Cooldownsin the Auto Scaling User Guide.

    
								AWS Management Console automatic scaling rule parameters for Amazon EMR.

Using the AWS Management Console to Configure Automatic Scaling

When you create a cluster, you configure a scaling policy for instance groups using the advanced cluster configuration options. You can also create or modify a scaling policy for an instance group in-service by modifying instance groups in the Hardware settings of an existing cluster.

  1. If you are creating a cluster, in the Amazon EMR console, select Create Cluster, select Go to advanced options, choose options for Step 1: Software and Steps, and then go to Step 2: Hardware Configuration.

    —or—

    If you are modifying an instance group in a running cluster, select your cluster from the cluster list, and then expand the Hardware section.

  2. Click the pencil icon that appears in the Auto Scaling column for the instance group you want to configure. If an automatic scaling policy is already configured for the instance group, the number of Maximum instances and Minimum instances appear in this column; otherwise, Not enabled appears.

    The Auto Scaling rules screen opens. Scale out and Scale in are selected by default, and default rules are pre-configured with settings suitable for many applications.

  3. Type the Maximum instances you want the instance group to contain after it scales out, and type the Minimum instances you want the instance group to contain after it scales in.

  4. Click the pencil to edit rule parameters, click the X to remove a rule from the policy, and click Add rule to add additional rules.

  5. Choose rule parameters as described earlier in this topic. For descriptions of available CloudWatch metrics for Amazon EMR, see Amazon EMR Metrics and Dimensions in the Amazon CloudWatch User Guide.

Using the AWS CLI to Configure Automatic Scaling

You can use AWS CLI commands for Amazon EMR to configure automatic scaling when you create a cluster and when you create an instance group. You can use a shorthand syntax, specifying the JSON configuration inline within the relevant commands, or you can reference a file containing the configuration JSON. You can also apply an automatic scaling policy to an existing instance group and remove an automatic scaling policy that was previously applied. In addition, you can retrieve details of a scaling policy configuration from a running cluster.

Important

When you create a cluster that has an automatic scaling policy, you must use the --auto-scaling-role EMR_AutoScaling_DefaultRole command. To implement automatic scaling on a cluster, the cluster must have the EMR_AutoScaling_DefaultRole. The role can only be added when the cluster is created, and cannot be added to an existing cluster.

For a detailed description of the parameters available when configuring an automatic scaling policy, see PutAutoScalingPolicy in Amazon EMR API Reference.

Creating a Cluster with an Automatic Scaling Policy Applied to an Instance Group

You can specify an automatic scaling configuration within the --instance-groups option of the aws emr create-cluster command. The following example illustrates a create-cluster command where an automatic scaling policy for the core instance group is provided inline. The command creates a scaling configuration equivalent to the default scale-out policy that appears when you create an auto scaling policy using the AWS Management Console for Amazon EMR. For brevity, a scale-in policy is not shown. Creating a scale-out rule without a scale-in rule is not recommended.

Copy
aws emr create-cluster --release-label emr-5.2.0 --service-role EMR_DefaultRole --ec2-attributes InstanceProfile=EMR_EC2_DefaultRole --auto-scaling-role EMR_AutoScaling_DefaultRole --instance-groups Name=MyMasterIG,InstanceGroupType=MASTER,InstanceType=m3.xlarge,InstanceCount=1 'Name=MyCoreIG,InstanceGroupType=CORE,InstanceType=m3.xlarge,InstanceCount=2,AutoScalingPolicy={Constraints={MinCapacity=2,MaxCapacity=10},Rules=[{Name=Default-scale-out,Description=Replicates the default scale-out rule in the console.,Action={SimpleScalingPolicyConfiguration={AdjustmentType=CHANGE_IN_CAPACITY,ScalingAdjustment=1,CoolDown=300}},Trigger={CloudWatchAlarmDefinition={ComparisonOperator=LESS_THAN,EvaluationPeriods=1,MetricName=YARNMemoryAvailablePercentage,Namespace=AWS/ElasticMapReduce,Period=300,Statistic=AVERAGE,Threshold=15,Unit=PERCENT,Dimensions=[{Key=JobFlowId,Value="${emr.clusterId}"}]}}}]}'

The following command illustrates using the command line to provide the automatic scaling policy definition as part of an instance group configuration file named instancegroupconfig.json.

Copy
aws emr create-cluster --release-label emr-5.2.0 --service-role EMR_DefaultRole --ec2-attributes InstanceProfile=EMR_EC2_DefaultRole --instance-groups file://your/path/to/instancegroupconfig.json --auto-scaling-role EMR_AutoScaling_DefaultRole

With the contents of the configuration file as follows:

Copy
[ { "InstanceCount": 1, "Name": "MyMasterIG", "InstanceGroupType": "MASTER", "InstanceType": "m3.xlarge" }, { "InstanceCount": 2, "Name": "MyCoreIG", "InstanceGroupType": "CORE", "InstanceType": "m3.xlarge", "AutoScalingPolicy": { "Constraints": { "MinCapacity": 2, "MaxCapacity": 10 }, "Rules": [ { "Name": "Default-scale-out", "Description": "Replicates the default scale-out rule in the console for YARN memory.", "Action":{ "SimpleScalingPolicyConfiguration":{ "AdjustmentType": "CHANGE_IN_CAPACITY", "ScalingAdjustment": 1, "CoolDown": 300 } }, "Trigger":{ "CloudWatchAlarmDefinition":{ "ComparisonOperator": "LESS_THAN", "EvaluationPeriods": 1, "MetricName": "YARNMemoryAvailablePercentage", "Namespace": "AWS/ElasticMapReduce", "Period": 300, "Threshold": 15, "Statistic": "AVERAGE", "Unit": "PERCENT", "Dimensions":[ { "Key" : "JobFlowId", "Value" : "${emr.clusterId}" } ] } } } ] } } ]

Adding an Instance Group with an Automatic Scaling Policy to a Cluster

You can specify a scaling policy configuration using the --instance-groups option with the add-instance-groups command in the same way you can when you use create-cluster. The following example uses a reference to a JSON file, instancegroupconfig.json, with the instance group configuration.

Copy
aws emr add-instance-groups --cluster-id j-1EKZ3TYEVF1S2 --instance-groups file://your/path/to/instancegroupconfig.json

Applying an Automatic Scaling Policy to an Existing Instance Group or Modifying an Applied Policy

Use the aws emr put-auto-scaling-policy command to apply an automatic scaling policy to an existing instance group. The instance group must be part of a cluster that uses the automatic scaling IAM role. The following example uses a reference to a JSON file, autoscaleconfig.json, that specifies the automatic scaling policy configuration.

Copy
aws emr put-auto-scaling-policy --cluster-id j-1EKZ3TYEVF1S2 --instance-group-id ig-3PLUZBA6WLS07 --auto-scaling-policy file://your/path/to/autoscaleconfig.json

The contents of the autoscaleconfig.json file, which defines the same scale-out rule as shown in the previous example, is shown below.

Copy
"AutoScalingPolicy": { "Constraints": { "MinCapacity": 2, "MaxCapacity": 10 }, "Rules": [ { "Name": "Default-scale-out", "Description": "Replicates the default scale-out rule in the console for YARN memory.", "Action":{ "SimpleScalingPolicyConfiguration":{ "AdjustmentType": "CHANGE_IN_CAPACITY", "ScalingAdjustment": 1, "CoolDown": 300 } }, "Trigger":{ "CloudWatchAlarmDefinition":{ "ComparisonOperator": "LESS_THAN", "EvaluationPeriods": 1, "MetricName": "YARNMemoryAvailablePercentage", "Namespace": "AWS/ElasticMapReduce", "Period": 300, "Threshold": 15, "Statistic": "AVERAGE", "Unit": "PERCENT", "Dimensions":[ { "Key" : "JobFlowId", "Value" : "${emr.clusterId}" } ] } } } ] }

Removing an Automatic Scaling Policy from an Instance Group

Copy
aws emr remove-auto-scaling-policy --cluster-id j-1EKZ3TYEVF1S2 --instance-group-id ig-3PLUZBA6WLS07

Retrieving an Automatic Scaling Policy Configuration

The describe-cluster command retrieves the policy configuration in the InstanceGroup block. For example, the following command retrieves the configuration for the cluster with a cluster ID of j-1CWOHP4PI30VJ.

Copy
aws emr describe-cluster –-cluster-id j-1CWOHP4PI30VJ

The command produces the following example output.

Copy
{ "Cluster": { "Configurations": [], "Id": "j-1CWOHP4PI30VJ", "NormalizedInstanceHours": 48, "Name": "Auto Scaling Cluster", "ReleaseLabel": "emr-5.2.0", "ServiceRole": "EMR_DefaultRole", "AutoTerminate": false, "TerminationProtected": true, "MasterPublicDnsName": "ec2-54-167-31-38.compute-1.amazonaws.com", "LogUri": "s3n://aws-logs-232939870606-us-east-1/elasticmapreduce/", "Ec2InstanceAttributes": { "Ec2KeyName": "performance", "AdditionalMasterSecurityGroups": [], "AdditionalSlaveSecurityGroups": [], "EmrManagedSlaveSecurityGroup": "sg-09fc9362", "Ec2AvailabilityZone": "us-east-1d", "EmrManagedMasterSecurityGroup": "sg-0bfc9360", "IamInstanceProfile": "EMR_EC2_DefaultRole" }, "Applications": [ { "Name": "Hadoop", "Version": "2.7.3" } ], "InstanceGroups": [ { "AutoScalingPolicy": { "Status": { "State": "ATTACHED", "StateChangeReason": { "Message": "" } }, "Constraints": { "MaxCapacity": 10, "MinCapacity": 2 }, "Rules": [ { "Name": "Default-scale-out", "Trigger": { "CloudWatchAlarmDefinition": { "MetricName": "YARNMemoryAvailablePercentage", "Unit": "PERCENT", "Namespace": "AWS/ElasticMapReduce", "Threshold": 15, "Dimensions": [ { "Key": "JobFlowId", "Value": "j-1CWOHP4PI30VJ" } ], "EvaluationPeriods": 1, "Period": 300, "ComparisonOperator": "LESS_THAN", "Statistic": "AVERAGE" } }, "Description": "", "Action": { "SimpleScalingPolicyConfiguration": { "CoolDown": 300, "AdjustmentType": "CHANGE_IN_CAPACITY", "ScalingAdjustment": 1 } } }, { "Name": "Default-scale-in", "Trigger": { "CloudWatchAlarmDefinition": { "MetricName": "YARNMemoryAvailablePercentage", "Unit": "PERCENT", "Namespace": "AWS/ElasticMapReduce", "Threshold": 0.75, "Dimensions": [ { "Key": "JobFlowId", "Value": "j-1CWOHP4PI30VJ" } ], "EvaluationPeriods": 1, "Period": 300, "ComparisonOperator": "GREATER_THAN", "Statistic": "AVERAGE" } }, "Description": "", "Action": { "SimpleScalingPolicyConfiguration": { "CoolDown": 300, "AdjustmentType": "CHANGE_IN_CAPACITY", "ScalingAdjustment": -1 } } } ] }, "Configurations": [], "InstanceType": "m3.xlarge", "Market": "ON_DEMAND", "Name": "Core - 2", "ShrinkPolicy": {}, "Status": { "Timeline": { "CreationDateTime": 1479413437.342, "ReadyDateTime": 1479413864.615 }, "State": "RUNNING", "StateChangeReason": { "Message": "" } }, "RunningInstanceCount": 2, "Id": "ig-3M16XBE8C3PH1", "InstanceGroupType": "CORE", "RequestedInstanceCount": 2, "EbsBlockDevices": [] }, { "Configurations": [], "Id": "ig-OP62I28NSE8M", "InstanceGroupType": "MASTER", "InstanceType": "m3.xlarge", "Market": "ON_DEMAND", "Name": "Master - 1", "ShrinkPolicy": {}, "EbsBlockDevices": [], "RequestedInstanceCount": 1, "Status": { "Timeline": { "CreationDateTime": 1479413437.342, "ReadyDateTime": 1479413752.088 }, "State": "RUNNING", "StateChangeReason": { "Message": "" } }, "RunningInstanceCount": 1 } ], "AutoScalingRole": "EMR_AutoScaling_DefaultRole", "Tags": [], "VisibleToAllUsers": true, "BootstrapActions": [], "Status": { "Timeline": { "CreationDateTime": 1479413437.339, "ReadyDateTime": 1479413863.666 }, "State": "WAITING", "StateChangeReason": { "Message": "Cluster ready after last step completed." } } } }