Menu
Amazon EMR
Management Guide

Create and Use IAM Roles for Amazon EMR

There are three ways to create IAM roles:

If you are using an IAM user and creating default roles for a cluster, your IAM user must have the following permissions:

  • iam:CreateRole

  • iam:PutRolePolicy

  • iam:CreateInstanceProfile

  • iam:AddRoleToInstanceProfile

  • iam:ListRoles

  • iam:GetPolicy

  • iam:GetInstanceProfile

  • iam:GetPolicyVersion

  • iam:AttachRolePolicy

  • iam:PassRole

The iam:PassRole permission allows cluster creation. The remaining permissions allow creation of the default roles.

Create and Use IAM Roles with the Amazon EMR Console

AWS customers whose accounts were created after release of Amazon EMR roles are required to specify an Amazon EMR (service) role and an EC2 instance profile in all regions when using the console. You can create default roles at cluster launch using the console, or you can specify other roles you may already be using.

Note

The EMR_AutoScaling_DefaultRole cannot be created using the console.

To create and use IAM roles with the console

  1. Open the Amazon EMR console at https://console.aws.amazon.com/elasticmapreduce/.

  2. Choose Create Cluster.

  3. In the Security and Access section, in the IAM Roles subsection, for Roles configuration, choose Default. If the default roles do not exist, they are created for you (assuming that you have appropriate permissions). If the roles exist, they are used for your cluster. After the roles are created, they are visible in the IAM console.

    Note

    To use custom roles with your cluster, choose Custom and select the existing roles from the list.

Create and Use IAM Roles with the AWS CLI

You can create the default Amazon EMR (service) role and EC2 instance profile using the CLI. After the roles are created, they are visible in the IAM console. If not already present, the roles are also auto-populated in the AWS CLI configuration file located at ~/.aws/config on Unix, Linux, and OS X systems or at C:\Users\USERNAME\.aws\config on Windows systems. After creation, you can use the default roles when you launch a cluster.

To create and use IAM roles with the AWS CLI

To create default roles using the AWS CLI, type the create-default-roles subcommand. To use the default roles at cluster launch, type the create-cluster subcommand with the --use-default-roles parameter. This command does not create the EMR_AutoScaling_DefaultRole.

  1. Type the following command to create default roles using the AWS CLI:

    Copy
    aws emr create-default-roles

    The output of the command lists the contents of the default Amazon EMR role, EMR_DefaultRole; the the default EC2 instance profile, EMR_EC2_DefaultRole. The AWS CLI configuration file is populated with these role names for the service_role and instance_profile values. For example, after this command, the configuration file might look like the following:

    Copy
    [default] output = json region = us-east-1 aws_access_key_id = myAccessKeyID aws_secret_access_key = mySecretAccessKey emr = service_role = EMR_DefaultRole instance_profile = EMR_EC2_DefaultRole

    You can also modify this file to use your own custom role and the AWS CLI uses that role by default.

  2. If the default roles already exist, you can use them when launching a cluster. Type the following command to use existing default roles when launching a cluster and replace myKey with the name of your EC2 key pair.

    Important

    This command will not add the EMR_AutoScaling_DefaultRole. You must explicitly add this role using the --auto-scaling-role EMR_AutoScaling_DefaultRole option with the create-cluster command. For more information, see Creating the IAM Role for Automatic Scaling.

    Copy
    aws emr create-cluster --name "Test cluster" --release-label emr-4.0.0 --applications Name=Hive Name=Pig --use-default-roles --ec2-attributes KeyName=myKey --instance-type m3.xlarge --instance-count 3

Attach Default IAM Roles to Managed Policies in the IAM Console

If you need to customize the default IAM roles, we recommend that you begin by creating the default roles, and then modify those roles as needed. In addition, if you created default roles before 6/11/2015, Amazon EMR did not attach managed policies to the role by default.

Use the following procedures to attach the IAM roles for Amazon EMR to their respective default managed policies. You can then view the default managed policies and edit them for customization as required.

Attach the Default Service Role to a Managed Policy for Amazon EMR

The EMR_DefaultRole consists of a role policy and a trust policy. You can view the most up-to-date AmazonElasticMapReduceRole in the Policies tab in the IAM console.

To attach EMR_DefaultRole to the AmazonElasticMapReduceRole managed policy using the console

Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/.

  1. Choose Policies, AmazonElasticMapReduceRole.

  2. Under Attached Entities, choose Attach, EMR_DefaultRole, Attach.

To attach EMR_DefaultRole to the AmazonElasticMapReduceRole managed policy using the AWS CLI

  • Use the following syntax to attach your pre-existing default role to the managed policy:

    Copy
    $ aws iam attach-role-policy --role-name EMR_DefaultRole\ --policy-arn arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole $ aws iam delete-role-policy --role-name EMR_DefaultRole --policy-name EMR_DefaultRole

Attach the Default EC2 Instance Profile to a Managed Policy for Amazon EMR

The default EC2 role consists of a role policy and a trust policy. You can view the most up-to-date AmazonElasticMapReduceforEC2Role in the Policies tab in the IAM console.

To attach EMR_EC2_DefaultRole to the managed policy using the console

Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/.

  1. Choose Policies, AmazonElasticMapReduceforEC2Role.

  2. Under Attached Entities, choose Attach, EMR_EC2_DefaultRole, and Attach.

To attach EMR_EC2_DefaultRole to the managed rolicy using the CLI

  • Use the following AWS CLI syntax to attach your pre-existing default role to the managed policy:

    Copy
    $ aws iam attach-role-policy --role-name EMR_EC2_DefaultRole\ --policy-arn arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role $ aws iam delete-role-policy --role-name EMR_EC2_DefaultRole --policy-name EMR_EC2_DefaultRole

Custom IAM Roles for Amazon EMR

If the default IAM roles provided by Amazon EMR do not meet your needs, you can create custom roles instead. For example, if your application does not access Amazon DynamoDB, you can leave out DynamoDB permissions in your custom IAM role.

For more information about creating and managing IAM roles, see the following topics in the IAM User Guide:

We recommend that you begin with the permissions in the managed policies AmazonElasticMapReduceforEC2Role and AmazonElasticMapReduceRole when developing custom IAM roles to use with Amazon EMR. You can then copy the contents of these default roles, create new IAM roles, paste in the copied permissions, and modify the pasted permissions.

The following is an example of a custom Amazon EC2 instance profile for use with Amazon EMR. This example is for a cluster that does not use Amazon RDS, or DynamoDB.

The access to Amazon SimpleDB is included to permit debugging from the console although in releases greater than 4.1, sqs:* is required for this purpose. Access to CloudWatch is included so the cluster can report metrics. Amazon SNS and Amazon SQS permissions are included for messaging. The minimum Amazon SQS permissions required for Amazon EMR are: sqs:SendMessage and sqs:QueueExist.

Copy
{ "Statement": [ { "Action": [ "cloudwatch:*", "ec2:Describe*", "elasticmapreduce:Describe*", "s3:*", "sdb:*", "sns:*", "sqs:*" ], "Effect": "Allow", "Resource": "*" } ] }

Important

The IAM role name and the instance profile name must match exactly when you use either the Amazon EMR console or CLI. The console shows IAM role names in the EC2 instance profile list. The CLI interprets the name as instance profile names. For this reason, if you use the IAM CLI or API to create an IAM role and its associated instance profile, we recommend that you give the new IAM role the same name as its associated instance profile. By default, if you create an IAM role with the IAM console and do not specify an instance profile name, the instance profile name is the same as the IAM role name.

In some situations, you might need to work with an IAM role whose associated instance profile does not have the same name as the role. This can occur if you use AWS CloudFormation to manage IAM roles for you, because AWS CloudFormation adds a suffix to the role name to create the instance profile name. In this case, you can use the Amazon EMR API or CLI to specify the instance profile name. Unlike the console, the API and CLI do not interpret an instance profile name as IAM role name. In the API, you can call the RunJobFlow action and pass the instance profile name for the JobFlowRole parameter. In the CLI, you can specify the instance profile name for the –-ec2-attributes InstanceProfile option of aws emr create-cluster command.