Amazon Elastic MapReduce
Developer Guide (API Version 2009-03-31)
Did this page help you?  Yes | No |  Tell us about it...
« PreviousNext »
View the PDF for this guide.Go to the AWS Discussion Forum for this product.Go to the Kindle Store to download this guide in Kindle format.

Configure IAM User Permissions

Amazon EMR supports AWS Identity and Access Management (IAM) policies. IAM is a web service that enables AWS customers to manage users and user permissions. For more information on IAM, go to Using IAM in the Using IAM guide.

IAM enables you to create users under your AWS account. You can define policies that limit the actions those users can take with your AWS resources. For example, you can choose to give an IAM user the ability to view, but not to create or delete, Amazon S3 buckets in your AWS account. IAM is available at no charge to all AWS account holders; you do not need to sign up for IAM. You can use IAM through the Amazon EMR console, the Amazon EMR CLI, and programmatically through the Amazon EMR API and the AWS SDKs.

Instead of giving permissions to individual users, it can be convenient to use IAM roles and group users with certain permissions. For more information, see Configure IAM Roles for Amazon EMR.

Configuring Cluster Access

By default, clusters created using the console and AWS CLI are visible to all IAM users. You can change this setting when you launch a cluster or after the cluster is created. If an IAM user launches a cluster, and that cluster is hidden from other IAM users on the AWS account, only that user will see the cluster. For example, if an IAM user uses the AWS CLI to run the list-clusters command, clusters created by other IAM users with IAM user access set to "No other IAM users" are not listed. This filtering occurs on all Amazon EMR interfaces—the console, CLI, API, and SDKs—and prevents IAM users from accessing and inadvertently changing clusters created by other IAM users. It is useful for clusters that are intended to be viewed by only a single IAM user and the main AWS account.

Note

This filtering does not prevent IAM users from viewing the underlying resources of the cluster, such as EC2 instances, by using AWS interfaces outside of Amazon EMR.

The default option, launching a cluster with IAM user access set to "All other IAM users," makes a cluster visible and accessible to all IAM users under a single AWS account. Using this feature, all IAM users on your account have access to the cluster and, by configuring the policies of the IAM groups they belong to, you control how those users interact with the cluster. For example, Devlin, a developer, belongs to a group that has an IAM policy that grants full access to all Amazon EMR functionality. He could launch a cluster that is visible to all other IAM users on his company's AWS account. A second IAM user, Ann, a data analyst with the same company, could then run queries on that cluster. Because Ann does not launch or terminate clusters, the IAM policy for the group she is in would only contain the permissions necessary for her to run her queries.

To configure cluster access using the Amazon EMR console

  1. Open the Amazon Elastic MapReduce console at https://console.aws.amazon.com/elasticmapreduce/.

  2. Click Create cluster.

  3. In the Security and Access section, in the IAM User Access field, choose All other IAM users to make the cluster visible and accessible to all IAM users on the AWS account (the default option). Choose No other IAM users to restrict access. For more information, see Configure IAM User Permissions.

    IAM user visibility
  4. Proceed with creating the cluster as described in Plan an Amazon EMR Cluster.

To configure cluster access using the AWS CLI

By default, clusters created using the AWS CLI are visible to all users. You may optionally specify the --visible-to-all-users parameter to make a cluster visible to all IAM users. To restrict cluster access on a new cluster using the AWS CLI, type the create-cluster command with the --no-visible-to-all-users parameter.

  • To restrict access to a new cluster, type the following command and replace myKey with the name of your EC2 key pair.

    • Linux, UNIX, and Mac OS X users:

      aws emr create-cluster --name "Test cluster" --ami-version 3.3 --no-visible-to-all-users \
      --applications Name=Hive Name=Pig Name=Hue \
      --use-default-roles --ec2-attributes KeyName=myKey \
      --instance-type m3.xlarge --instance-count 3 
    • Windows users:

      aws emr create-cluster --name "Test cluster" --ami-version 3.3 --no-visible-to-all-users --applications Name=Hive Name=Pig Name=Hue --use-default-roles --ec2-attributes KeyName=myKey --instance-type m3.xlarge --instance-count 3

    When you specify the instance count without using the --instance-groups parameter, a single Master node is launched, and the remaining instances are launched as core nodes. All nodes will use the instance type specified in the command.

    Note

    If you have not previously created the default EMR service role and EC2 instance profile, type aws emr create-default-roles to create them before typing the create-cluster subcommand.

    If you are configuring cluster access to an existing cluster, type the modify-cluster-attributes subcommand with the --no-visible-to-all-users parameter or the --visible-to-all-users parameter. The visibility of a running cluster can be changed only by the IAM user that created the cluster or the AWS account owner. To use this subcommand, you need the cluster identifier available via the console or the list-clusters subcommand.

    To restrict access to a running cluster, type the following command.

    aws emr modify-cluster-attributes --cluster-id j-1GMZXXXXXXYMZ --no-visible-to-all-users

    For more information on using Amazon EMR commands in the AWS CLI, see http://docs.aws.amazon.com/cli/latest/reference/emr.

To configure cluster access using the Amazon EMR CLI

  • By default, clusters created using the Amazon EMR CLI are not visible to all users. If you are adding IAM user visibility to a new cluster using the Amazon EMR CLI, add the --visible-to-all-users flag to the cluster call as shown in the following example.

    In the directory where you installed the Amazon EMR CLI, type the following command. For more information, see the Command Line Interface Reference for Amazon EMR.

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --create --alive /
      --instance-type m1.xlarge --num-instances 2 /
      --visible-to-all-users 
    • Windows users:

      ruby elastic-mapreduce --create --alive --instance-type m1.xlarge --num-instances 2 --visible-to-all-users 

    If you are adding IAM user visibility to an existing cluster, you can use the --set-visible-to-all-users option of the Amazon EMR CLI, and specify identifier of the cluster to modify. This is shown in the following example, where JobFlowId would be replaced by the cluster identifier of your cluster. The visibility of a running cluster can be changed only by the IAM user that created the cluster or the AWS account that owns the cluster.

    In the directory where you installed the Amazon EMR CLI, type the following command. For more information, see the Command Line Interface Reference for Amazon EMR.

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --set-visible-to-all-users true --jobflow JobFlowId
    • Windows users:

      ruby elastic-mapreduce --set-visible-to-all-users true --jobflow JobFlowId

To configure cluster access using the Amazon EMR API

  • To configure IAM user access on a new cluster, call RunJobFlow and set VisibleToAllUsers=true or false, as shown in the following example.

    https://elasticmapreduce.amazonaws.com?Operation=RunJobFlow
    &Name=MyJobFlowName
    &VisibleToAllUsers=true
    &LogUri=s3n%3A%2F%2Fmybucket%2Fsubdir
    &Instances.MasterInstanceType=m1.large
    &Instances.SlaveInstanceType=m1.large
    &Instances.InstanceCount=4
    &Instances.Ec2KeyName=myec2keyname
    &Instances.Placement.AvailabilityZone=us-east-1a
    &Instances.KeepJobFlowAliveWhenNoSteps=true
    &Instances.TerminationProtected=true
    &Steps.member.1.Name=MyStepName
    &Steps.member.1.ActionOnFailure=CONTINUE
    &Steps.member.1.HadoopJarStep.Jar=MyJarFile
    &Steps.member.1.HadoopJarStep.MainClass=MyMainClass
    &Steps.member.1.HadoopJarStep.Args.member.1=arg1
    &Steps.member.1.HadoopJarStep.Args.member.2=arg2
    &AuthParams