AWS::EMR::InstanceGroupConfig - AWS CloudFormation

AWS::EMR::InstanceGroupConfig

Use InstanceGroupConfig to define instance groups for an EMR cluster. A cluster can not use both instance groups and instance fleets. For more information, see Create a Cluster with Instance Fleets or Uniform Instance Groups in the Amazon EMR Management Guide.

Important

You can currently only add task instance groups to a cluster with this resource. If you use this resource, CloudFormation waits for the cluster launch to complete before adding the task instance group to the cluster. In order to add task instance groups to the cluster as part of the cluster launch and minimize delays in provisioning task nodes, use the TaskInstanceGroups subproperty for the AWS::EMR::Cluster JobFlowInstancesConfig property instead. To use this subproperty, see AWS::EMR::Cluster for examples.

Syntax

To declare this entity in your AWS CloudFormation template, use the following syntax:

JSON

{ "Type" : "AWS::EMR::InstanceGroupConfig", "Properties" : { "AutoScalingPolicy" : AutoScalingPolicy, "BidPrice" : String, "Configurations" : [ Configuration, ... ], "CustomAmiId" : String, "EbsConfiguration" : EbsConfiguration, "InstanceCount" : Integer, "InstanceRole" : String, "InstanceType" : String, "JobFlowId" : String, "Market" : String, "Name" : String } }

YAML

Type: AWS::EMR::InstanceGroupConfig Properties: AutoScalingPolicy: AutoScalingPolicy BidPrice: String Configurations: - Configuration CustomAmiId: String EbsConfiguration: EbsConfiguration InstanceCount: Integer InstanceRole: String InstanceType: String JobFlowId: String Market: String Name: String

Properties

AutoScalingPolicy

AutoScalingPolicy is a subproperty of InstanceGroupConfig. AutoScalingPolicy defines how an instance group dynamically adds and terminates EC2 instances in response to the value of a CloudWatch metric. For more information, see Using Automatic Scaling in Amazon EMR in the Amazon EMR Management Guide.

Required: No

Type: AutoScalingPolicy

Update requires: No interruption

BidPrice

If specified, indicates that the instance group uses Spot Instances. This is the maximum price you are willing to pay for Spot Instances. Specify OnDemandPrice to set the amount equal to the On-Demand price, or specify an amount in USD.

Required: No

Type: String

Pattern: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*

Minimum: 0

Maximum: 256

Update requires: Replacement

Configurations
Note

Amazon EMR releases 4.x or later.

The list of configurations supplied for an Amazon EMR cluster instance group. You can specify a separate configuration for each instance group (master, core, and task).

Required: No

Type: Array of Configuration

Update requires: Replacement

CustomAmiId

The custom AMI ID to use for the provisioned instance group.

Required: No

Type: String

Pattern: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*

Minimum: 0

Maximum: 256

Update requires: Replacement

EbsConfiguration

EbsConfiguration determines the EBS volumes to attach to EMR cluster instances.

Required: No

Type: EbsConfiguration

Update requires: Replacement

InstanceCount

Target number of instances for the instance group.

Required: Yes

Type: Integer

Update requires: No interruption

InstanceRole

The role of the instance group in the cluster.

Allowed Values: TASK

Required: Yes

Type: String

Update requires: Replacement

InstanceType

The Amazon EC2 instance type for all instances in the instance group.

Required: Yes

Type: String

Pattern: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*

Minimum: 1

Maximum: 256

Update requires: Replacement

JobFlowId

The ID of an Amazon EMR cluster that you want to associate this instance group with.

Required: Yes

Type: String

Update requires: Replacement

Market

Market type of the Amazon EC2 instances used to create a cluster node.

Required: No

Type: String

Allowed values: ON_DEMAND | SPOT

Update requires: Replacement

Name

Friendly name given to the instance group.

Required: No

Type: String

Pattern: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*

Minimum: 0

Maximum: 256

Update requires: Replacement

Return values

Ref

When you pass the logical ID of this resource to the intrinsic Ref function, Ref returns the ID of the instance group.

For more information about using the Ref function, see Ref.

Fn::GetAtt

Examples

Example instance group configuration specifications.

Add a task instance group

The following example adds a task instance group to a cluster named TestCluster.

JSON

"TestInstanceGroupConfig": { "Type": "AWS: : EMR: : InstanceGroupConfig", "Properties": { "InstanceCount": 2, "InstanceType": "m3.xlarge", "InstanceRole": "TASK", "Market": "ON_DEMAND", "Name": "cfnTask2", "JobFlowId": { "Ref": "cluster" } } }

YAML

TestInstanceGroupConfig: Properties: InstanceCount: 2 InstanceRole: TASK InstanceType: m3.xlarge JobFlowId: Ref: cluster Market: ON_DEMAND Name: cfnTask2 Type: "AWS::EMR::InstanceGroupConfig"

Specify an Automatic Scaling Policy

The following example defines an AutoScalingPolicy for an InstanceGroupConfig resource.

JSON

"MyInstanceGroupConfig": { "Type": "AWS::EMR::InstanceGroupConfig", "Properties": { "InstanceCount": 1, "InstanceType": { "Ref": "InstanceType" }, "InstanceRole": "TASK", "Market": "ON_DEMAND", "Name": "cfnTask", "JobFlowId": { "Ref": "MyCluster" }, "AutoScalingPolicy": { "Constraints": { "MinCapacity": { "Ref": "MinCapacity" }, "MaxCapacity": { "Ref": "MaxCapacity" } }, "Rules": [ { "Name": "Scale-out", "Description": "Scale-out policy", "Action": { "SimpleScalingPolicyConfiguration": { "AdjustmentType": "CHANGE_IN_CAPACITY", "ScalingAdjustment": 1, "CoolDown": 300 } }, "Trigger": { "CloudWatchAlarmDefinition": { "Dimensions": [ { "Key": "JobFlowId", "Value": "${emr.clusterId}" } ], "EvaluationPeriods": 1, "Namespace": "AWS/ElasticMapReduce", "Period": 300, "ComparisonOperator": "LESS_THAN", "Statistic": "AVERAGE", "Threshold": 15, "Unit": "PERCENT", "MetricName": "YARNMemoryAvailablePercentage" } } }, { "Name": "Scale-in", "Description": "Scale-in policy", "Action": { "SimpleScalingPolicyConfiguration": { "AdjustmentType": "CHANGE_IN_CAPACITY", "ScalingAdjustment": -1, "CoolDown": 300 } }, "Trigger": { "CloudWatchAlarmDefinition": { "Dimensions": [ { "Key": "JobFlowId", "Value": "${emr.clusterId}" } ], "EvaluationPeriods": 1, "Namespace": "AWS/ElasticMapReduce", "Period": 300, "ComparisonOperator": "GREATER_THAN", "Statistic": "AVERAGE", "Threshold": 75, "Unit": "PERCENT", "MetricName": "YARNMemoryAvailablePercentage" } } } ] } } }

YAML

MyInstanceGroupConfig: Type: 'AWS::EMR::InstanceGroupConfig' Properties: InstanceCount: 1 InstanceType: !Ref InstanceType InstanceRole: TASK Market: ON_DEMAND Name: cfnTask JobFlowId: !Ref MyCluster AutoScalingPolicy: Constraints: MinCapacity: !Ref MinCapacity MaxCapacity: !Ref MaxCapacity Rules: - Name: Scale-out Description: Scale-out policy Action: SimpleScalingPolicyConfiguration: AdjustmentType: CHANGE_IN_CAPACITY ScalingAdjustment: 1 CoolDown: 300 Trigger: CloudWatchAlarmDefinition: Dimensions: - Key: JobFlowId Value: '${emr.clusterId}' EvaluationPeriods: 1 Namespace: AWS/ElasticMapReduce Period: 300 ComparisonOperator: LESS_THAN Statistic: AVERAGE Threshold: 15 Unit: PERCENT MetricName: YARNMemoryAvailablePercentage - Name: Scale-in Description: Scale-in policy Action: SimpleScalingPolicyConfiguration: AdjustmentType: CHANGE_IN_CAPACITY ScalingAdjustment: -1 CoolDown: 300 Trigger: CloudWatchAlarmDefinition: Dimensions: - Key: JobFlowId Value: '${emr.clusterId}' EvaluationPeriods: 1 Namespace: AWS/ElasticMapReduce Period: 300 ComparisonOperator: GREATER_THAN Statistic: AVERAGE Threshold: 75 Unit: PERCENT MetricName: YARNMemoryAvailablePercentage