AWS CloudFormation
User Guide (Version )

AWS::EMR::Cluster

The AWS::EMR::Cluster resource specifies an Amazon EMR cluster. This cluster is a collection of Amazon EC2 instances that run open source big data frameworks and applications to process and analyze vast amounts of data. For more information, see the Amazon EMR Management Guide.

Syntax

To declare this entity in your AWS CloudFormation template, use the following syntax:

JSON

{ "Type" : "AWS::EMR::Cluster", "Properties" : { "AdditionalInfo" : Json, "Applications" : [ Application, ... ], "AutoScalingRole" : String, "BootstrapActions" : [ BootstrapActionConfig, ... ], "Configurations" : [ Configuration, ... ], "CustomAmiId" : String, "EbsRootVolumeSize" : Integer, "Instances" : JobFlowInstancesConfig, "JobFlowRole" : String, "KerberosAttributes" : KerberosAttributes, "LogUri" : String, "Name" : String, "ReleaseLabel" : String, "ScaleDownBehavior" : String, "SecurityConfiguration" : String, "ServiceRole" : String, "Steps" : [ StepConfig, ... ], "Tags" : [ Tag, ... ], "VisibleToAllUsers" : Boolean } }

Properties

AdditionalInfo

A JSON string for selecting additional features.

Required: No

Type: Json

Minimum: 0

Maximum: 10280

Pattern: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*

Update requires: Replacement

Applications

The applications to install on this cluster, for example, Spark, Flink, Oozie, Zeppelin, and so on.

Required: No

Type: List of Application

Update requires: Replacement

AutoScalingRole

An IAM role for automatic scaling policies. The default role is EMR_AutoScaling_DefaultRole. The IAM role provides permissions that the automatic scaling feature requires to launch and terminate EC2 instances in an instance group.

Required: No

Type: String

Minimum: 0

Maximum: 10280

Pattern: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*

Update requires: Replacement

BootstrapActions

A list of bootstrap actions to run before Hadoop starts on the cluster nodes.

Required: No

Type: List of BootstrapActionConfig

Update requires: Replacement

Configurations

Applies only to Amazon EMR releases 4.x and later. The list of Configurations supplied to the EMR cluster.

Required: No

Type: List of Configuration

Update requires: Replacement

CustomAmiId

Available only in Amazon EMR version 5.7.0 and later. The ID of a custom Amazon EBS-backed Linux AMI if the cluster uses a custom AMI.

Required: No

Type: String

Minimum: 0

Maximum: 256

Pattern: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*

Update requires: Replacement

EbsRootVolumeSize

The size, in GiB, of the EBS root device volume of the Linux AMI that is used for each EC2 instance. Available in Amazon EMR version 4.x and later.

Required: No

Type: Integer

Update requires: Replacement

Instances

A specification of the number and type of Amazon EC2 instances.

Required: Yes

Type: JobFlowInstancesConfig

Update requires: Some interruptions

JobFlowRole

Also called instance profile and EC2 role. An IAM role for an EMR cluster. The EC2 instances of the cluster assume this role. The default role is EMR_EC2_DefaultRole. In order to use the default role, you must have already created it using the CLI or console.

Required: Yes

Type: String

Minimum: 0

Maximum: 10280

Pattern: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*

Update requires: Replacement

KerberosAttributes

Attributes for Kerberos configuration when Kerberos authentication is enabled using a security configuration. For more information see Use Kerberos Authentication in the EMR Management Guide.

Required: No

Type: KerberosAttributes

Update requires: Replacement

LogUri

The path to the Amazon S3 location where logs for this cluster are stored.

Required: No

Type: String

Update requires: Replacement

Name

The name of the cluster.

Required: Yes

Type: String

Update requires: Replacement

ReleaseLabel

The Amazon EMR release label, which determines the version of open-source application packages installed on the cluster. Release labels are in the form emr-x.x.x, where x.x.x is an Amazon EMR release version, for example, emr-5.14.0. For more information about Amazon EMR release versions and included application versions and features, see https://docs.aws.amazon.com/emr/latest/ReleaseGuide/. The release label applies only to Amazon EMR releases versions 4.x and later. Earlier versions use AmiVersion.

Required: No

Type: String

Update requires: Replacement

ScaleDownBehavior

The way that individual Amazon EC2 instances terminate when an automatic scale-in activity occurs or an instance group is resized. TERMINATE_AT_INSTANCE_HOUR indicates that Amazon EMR terminates nodes at the instance-hour boundary, regardless of when the request to terminate the instance was submitted. This option is only available with Amazon EMR 5.1.0 and later and is the default for clusters created using that version. TERMINATE_AT_TASK_COMPLETION indicates that Amazon EMR blacklists and drains tasks from nodes before terminating the Amazon EC2 instances, regardless of the instance-hour boundary. With either behavior, Amazon EMR removes the least active nodes first and blocks instance termination if it could lead to HDFS corruption. TERMINATE_AT_TASK_COMPLETION is available only in Amazon EMR version 4.1.0 and later, and is the default for versions of Amazon EMR earlier than 5.1.0.

Required: No

Type: String

Allowed Values: TERMINATE_AT_INSTANCE_HOUR | TERMINATE_AT_TASK_COMPLETION

Update requires: Replacement

SecurityConfiguration

The name of the security configuration applied to the cluster.

Required: No

Type: String

Minimum: 0

Maximum: 10280

Pattern: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*

Update requires: Replacement

ServiceRole

The IAM role that will be assumed by the Amazon EMR service to access AWS resources on your behalf.

Required: Yes

Type: String

Update requires: Replacement

Steps

A list of steps to run.

Required: No

Type: List of StepConfig

Update requires: Replacement

Tags

A list of tags associated with a cluster.

Required: No

Type: List of Tag

Update requires: No interruption

VisibleToAllUsers

Indicates whether the cluster is visible to all IAM users of the AWS account associated with the cluster. If this value is set to true, all IAM users of that AWS account can view and manage the cluster if they have the proper policy permissions set. If this value is false, only the IAM user that created the cluster can view and manage it. This value can be changed using the SetVisibleToAllUsers action.

Required: No

Type: Boolean

Update requires: No interruption

Return Values

Ref

When you pass the logical ID of this resource to the intrinsic Ref function, Ref returns returns the cluster ID, such as j-1ABCD123AB1A.

For more information about using the Ref function, see Ref.

Fn::GetAtt

The Fn::GetAtt intrinsic function returns a value for a specified attribute of this type. The following are the available attributes and sample return values.

For more information about using the Fn::GetAtt intrinsic function, see Fn::GetAtt.

MasterPublicDNS

The public DNS name of the master node (instance), such as ec2-12-123-123-123.us-west-2.compute.amazonaws.com.

Examples

Create cluster examples.

Create a Cluster Using a Custom AMI for EC2 Instances

The following example template specifies a cluster using a custom Amazon Linux AMI for the EC2 instances in the cluster.

JSON

{ "AWSTemplateFormatVersion": "2010-09-09", "Parameters" : { "CustomAmiId" : { "Type" : "String" }, "InstanceType" : { "Type" : "String" }, "ReleaseLabel" : { "Type" : "String" }, "SubnetId" : { "Type" : "String" }, "TerminationProtected" : { "Type" : "String", "Default" : "false" }, "ElasticMapReducePrincipal" : { "Type" : "String" }, "Ec2Principal" : { "Type" : "String" } }, "Resources": { "cluster": { "Type": "AWS::EMR::Cluster", "Properties": { "CustomAmiId" : {"Ref" : "CustomAmiId"}, "Instances": { "MasterInstanceGroup": { "InstanceCount": 1, "InstanceType": {"Ref" : "InstanceType"}, "Market": "ON_DEMAND", "Name": "cfnMaster" }, "CoreInstanceGroup": { "InstanceCount": 1, "InstanceType": {"Ref" : "InstanceType"}, "Market": "ON_DEMAND", "Name": "cfnCore" }, "TerminationProtected" : {"Ref" : "TerminationProtected"}, "Ec2SubnetId" : {"Ref" : "SubnetId"} }, "Name": "CFNtest", "JobFlowRole" : {"Ref": "emrEc2InstanceProfile"}, "ServiceRole" : {"Ref": "emrRole"}, "ReleaseLabel" : {"Ref" : "ReleaseLabel"}, "VisibleToAllUsers" : true, "Tags": [ { "Key": "key1", "Value": "value1" } ] } }, "emrRole": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2008-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": {"Ref" : "ElasticMapReducePrincipal"} }, "Action": "sts:AssumeRole" } ] }, "Path": "/", "ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole"] } }, "emrEc2Role": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2008-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": {"Ref" : "Ec2Principal"} }, "Action": "sts:AssumeRole" } ] }, "Path": "/", "ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role"] } }, "emrEc2InstanceProfile": { "Type": "AWS::IAM::InstanceProfile", "Properties": { "Path": "/", "Roles": [ { "Ref": "emrEc2Role" } ] } } } }

YAML

AWSTemplateFormatVersion: 2010-09-09 Parameters: CustomAmiId: Type: String InstanceType: Type: String ReleaseLabel: Type: String SubnetId: Type: String TerminationProtected: Type: String Default: 'false' ElasticMapReducePrincipal: Type: String Ec2Principal: Type: String Resources: cluster: Type: AWS::EMR::Cluster Properties: CustomAmiId: !Ref CustomAmiId Instances: MasterInstanceGroup: InstanceCount: 1 InstanceType: !Ref InstanceType Market: ON_DEMAND Name: cfnMaster CoreInstanceGroup: InstanceCount: 1 InstanceType: !Ref InstanceType Market: ON_DEMAND Name: cfnCore TerminationProtected: !Ref TerminationProtected Ec2SubnetId: !Ref SubnetId Name: CFNtest JobFlowRole: !Ref emrEc2InstanceProfile ServiceRole: !Ref emrRole ReleaseLabel: !Ref ReleaseLabel VisibleToAllUsers: true Tags: - Key: key1 Value: value1 emrRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2008-10-17 Statement: - Sid: '' Effect: Allow Principal: Service: !Ref ElasticMapReducePrincipal Action: 'sts:AssumeRole' Path: / ManagedPolicyArns: - 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole' emrEc2Role: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2008-10-17 Statement: - Sid: '' Effect: Allow Principal: Service: !Ref Ec2Principal Action: 'sts:AssumeRole' Path: / ManagedPolicyArns: - 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role' emrEc2InstanceProfile: Type: AWS::IAM::InstanceProfile Properties: Path: / Roles: - !Ref emrEc2Role

Create a Cluster and Specify the Root Volume Size of EC2 Instances

The following example template enables you to specify the size of the EBS root volume for cluster instances.

JSON

{ "AWSTemplateFormatVersion": "2010-09-09", "Parameters" : { "InstanceType" : { "Type" : "String" }, "ReleaseLabel" : { "Type" : "String" }, "SubnetId" : { "Type" : "String" }, "TerminationProtected" : { "Type" : "String", "Default" : "false" }, "EbsRootVolumeSize" : { "Type" : "String" } }, "Resources": { "cluster": { "Type": "AWS::EMR::Cluster", "Properties": { "EbsRootVolumeSize" : {"Ref" : "EbsRootVolumeSize"}, "Instances": { "MasterInstanceGroup": { "InstanceCount": 1, "InstanceType": {"Ref" : "InstanceType"}, "Market": "ON_DEMAND", "Name": "cfnMaster" }, "CoreInstanceGroup": { "InstanceCount": 1, "InstanceType": {"Ref" : "InstanceType"}, "Market": "ON_DEMAND", "Name": "cfnCore" }, "TerminationProtected" : {"Ref" : "TerminationProtected"}, "Ec2SubnetId" : {"Ref" : "SubnetId"} }, "Name": "CFNtest", "JobFlowRole" : {"Ref": "emrEc2InstanceProfile"}, "ServiceRole" : {"Ref": "emrRole"}, "ReleaseLabel" : {"Ref" : "ReleaseLabel"}, "VisibleToAllUsers" : true, "Tags": [ { "Key": "key1", "Value": "value1" } ] } }, "emrRole": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2008-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "elasticmapreduce.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }, "Path": "/", "ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole"] } }, "emrEc2Role": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2008-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "ec2.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }, "Path": "/", "ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role"] } }, "emrEc2InstanceProfile": { "Type": "AWS::IAM::InstanceProfile", "Properties": { "Path": "/", "Roles": [ { "Ref": "emrEc2Role" } ] } } } }

YAML

AWSTemplateFormatVersion: 2010-09-09 Parameters: InstanceType: Type: String ReleaseLabel: Type: String SubnetId: Type: String TerminationProtected: Type: String Default: 'false' EbsRootVolumeSize: Type: String Resources: cluster: Type: AWS::EMR::Cluster Properties: EbsRootVolumeSize: !Ref EbsRootVolumeSize Instances: MasterInstanceGroup: InstanceCount: 1 InstanceType: !Ref InstanceType Market: ON_DEMAND Name: cfnMaster CoreInstanceGroup: InstanceCount: 1 InstanceType: !Ref InstanceType Market: ON_DEMAND Name: cfnCore TerminationProtected: !Ref TerminationProtected Ec2SubnetId: !Ref SubnetId Name: CFNtest JobFlowRole: !Ref emrEc2InstanceProfile ServiceRole: !Ref emrRole ReleaseLabel: !Ref ReleaseLabel VisibleToAllUsers: true Tags: - Key: key1 Value: value1 emrRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2008-10-17 Statement: - Sid: '' Effect: Allow Principal: Service: elasticmapreduce.amazonaws.com Action: 'sts:AssumeRole' Path: / ManagedPolicyArns: - 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole' emrEc2Role: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2008-10-17 Statement: - Sid: '' Effect: Allow Principal: Service: ec2.amazonaws.com Action: 'sts:AssumeRole' Path: / ManagedPolicyArns: - 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role' emrEc2InstanceProfile: Type: AWS::IAM::InstanceProfile Properties: Path: / Roles: - !Ref emrEc2Role

Create a Cluster with Kerberos Authentication

The following example template enables you to specify the Kerberos authentication configuration for an EMR cluster.

JSON

{ "AWSTemplateFormatVersion": "2010-09-09", "Parameters" : { "CrossRealmTrustPrincipalPassword" : { "Type" : "String" }, "KdcAdminPassword" : { "Type" : "String" }, "Realm" : { "Type" : "String" }, "InstanceType" : { "Type" : "String" }, "ReleaseLabel" : { "Type" : "String" }, "SubnetId" : { "Type" : "String" } }, "Resources": { "cluster": { "Type": "AWS::EMR::Cluster", "Properties": { "Instances": { "MasterInstanceGroup": { "InstanceCount": 1, "InstanceType": {"Ref" : "InstanceType"}, "Market": "ON_DEMAND", "Name": "cfnMaster" }, "CoreInstanceGroup": { "InstanceCount": 1, "InstanceType": {"Ref" : "InstanceType"}, "Market": "ON_DEMAND", "Name": "cfnCore" }, "Ec2SubnetId" : {"Ref" : "SubnetId"} }, "Name": "CFNtest2", "JobFlowRole" : {"Ref": "emrEc2InstanceProfile"}, "KerberosAttributes" : { "CrossRealmTrustPrincipalPassword" : "CfnIntegrationTest-1", "KdcAdminPassword" : "CfnIntegrationTest-1", "Realm": "EC2.INTERNAL" }, "ServiceRole" : {"Ref": "emrRole"}, "ReleaseLabel" : {"Ref" : "ReleaseLabel"}, "SecurityConfiguration" : {"Ref" : "securityConfiguration"}, "VisibleToAllUsers" : true, "Tags": [ { "Key": "key1", "Value": "value1" } ] } }, "key" : { "Type" : "AWS::KMS::Key", "Properties" : { "KeyPolicy" : { "Version": "2012-10-17", "Id": "key-default-1", "Statement": [ { "Sid": "Enable IAM User Permissions", "Effect": "Allow", "Principal": { "AWS": { "Fn::GetAtt" : ["emrEc2Role", "Arn"]} }, "Action": "kms:*", "Resource": "*" }, { "Sid": "Enable IAM User Permissions", "Effect": "Allow", "Principal": { "AWS": { "Fn::Join" : ["" , ["arn:aws:iam::", {"Ref" : "AWS::AccountId"} ,":root" ]] } }, "Action": "kms:*", "Resource": "*" } ] } } }, "securityConfiguration": { "Type" : "AWS::EMR::SecurityConfiguration", "Properties" : { "SecurityConfiguration" : { "AuthenticationConfiguration": { "KerberosConfiguration": { "Provider": "ClusterDedicatedKdc", "ClusterDedicatedKdcConfiguration": { "TicketLifetimeInHours": 24, "CrossRealmTrustConfiguration": { "Realm": "AD.DOMAIN.COM", "Domain": "ad.domain.com", "AdminServer": "ad.domain.com", "KdcServer": "ad.domain.com" } } } } } } }, "emrRole": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2008-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "elasticmapreduce.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }, "Path": "/", "ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole"] } }, "emrEc2Role": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2008-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "ec2.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }, "Path": "/", "ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role"] } }, "emrEc2InstanceProfile": { "Type": "AWS::IAM::InstanceProfile", "Properties": { "Path": "/", "Roles": [ { "Ref": "emrEc2Role" } ] } } }, "Outputs" : { "keyArn" : { "Value" : {"Fn::GetAtt" : ["key", "Arn"]} } } }

YAML

AWSTemplateFormatVersion: 2010-09-09 Parameters: CrossRealmTrustPrincipalPassword: Type: String KdcAdminPassword: Type: String Realm: Type: String InstanceType: Type: String ReleaseLabel: Type: String SubnetId: Type: String Resources: cluster: Type: 'AWS::EMR::Cluster' Properties: Instances: MasterInstanceGroup: InstanceCount: 1 InstanceType: !Ref InstanceType Market: ON_DEMAND Name: cfnMaster CoreInstanceGroup: InstanceCount: 1 InstanceType: !Ref InstanceType Market: ON_DEMAND Name: cfnCore Ec2SubnetId: !Ref SubnetId Name: CFNtest2 JobFlowRole: !Ref emrEc2InstanceProfile KerberosAttributes: CrossRealmTrustPrincipalPassword: CfnIntegrationTest-1 KdcAdminPassword: CfnIntegrationTest-1 Realm: EC2.INTERNAL ServiceRole: !Ref emrRole ReleaseLabel: !Ref ReleaseLabel SecurityConfiguration: !Ref securityConfiguration VisibleToAllUsers: true Tags: - Key: key1 Value: value1 key: Type: 'AWS::KMS::Key' Properties: KeyPolicy: Version: 2012-10-17 Id: key-default-1 Statement: - Sid: Enable IAM User Permissions Effect: Allow Principal: AWS: !GetAtt - emrEc2Role - Arn Action: 'kms:*' Resource: '*' - Sid: Enable IAM User Permissions Effect: Allow Principal: AWS: !Join - '' - - 'arn:aws:iam::' - !Ref 'AWS::AccountId' - ':root' Action: 'kms:*' Resource: '*' securityConfiguration: Type: 'AWS::EMR::SecurityConfiguration' Properties: SecurityConfiguration: AuthenticationConfiguration: KerberosConfiguration: Provider: ClusterDedicatedKdc ClusterDedicatedKdcConfiguration: TicketLifetimeInHours: 24 CrossRealmTrustConfiguration: Realm: AD.DOMAIN.COM Domain: ad.domain.com AdminServer: ad.domain.com KdcServer: ad.domain.com emrRole: Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: Version: 2008-10-17 Statement: - Sid: '' Effect: Allow Principal: Service: elasticmapreduce.amazonaws.com Action: 'sts:AssumeRole' Path: / ManagedPolicyArns: - 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole' emrEc2Role: Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: Version: 2008-10-17 Statement: - Sid: '' Effect: Allow Principal: Service: ec2.amazonaws.com Action: 'sts:AssumeRole' Path: / ManagedPolicyArns: - 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role' emrEc2InstanceProfile: Type: 'AWS::IAM::InstanceProfile' Properties: Path: / Roles: - !Ref emrEc2Role Outputs: keyArn: Value: !GetAtt - key - Arn