Select a Amazon VPC Subnet for the Cluster (Optional)

To allow users to launch clusters in a VPC without the ability to modify the VPC

1. Create the VPC and launch Amazon EMR into a subnet of that VPC using an account with permissions to administer Amazon VPC and Amazon EMR.

2. Create a second user account with permissions to call the RunJobFlow, DescribeJobFlows, TerminateJobFlows, and AddJobFlowStep actions in the Amazon EMR API. You should also create an IAM policy that allows this user to launch EC2 instances. An example of this is shown below.

   {
     "Statement": [
       {
         "Action": [
   	     "ec2:AuthorizeSecurityGroupIngress",
   	     "ec2:CancelSpotInstanceRequests",
   	     "ec2:CreateSecurityGroup",
   	     "ec2:CreateTags",
   	     "ec2:DescribeAvailabilityZones",
   	     "ec2:DescribeInstances",
   	     "ec2:DescribeSecurityGroups",
   	     "ec2:DescribeSpotInstanceRequests",
   	     "ec2:ModifyImageAttribute",	
   	     "ec2:ModifyInstanceAttribute",
   	     "ec2:RequestSpotInstances",
   	     "ec2:RunInstances",
   	     "ec2:TerminateInstances"  
         ],
         "Effect": "Allow",
         "Resource": "*"
       },
       {
         "Action": [
            "elasticmapreduce:AddInstanceGroups",
            "elasticmapreduce:AddJobFlowSteps", 
            "elasticmapreduce:DescribeJobFlows",
            "elasticmapreduce:ModifyInstanceGroups",
            "elasticmapreduce:RunJobFlow"
   	     "elasticmapreduce:TerminateJobFlows"
         ],
         "Effect": "Allow",
         "Resource": "*"
       }
     
     }
   				

   Users with the IAM permissions set above are able to launch clusters within the VPC subnet, but are not able to change the VPC configuration.

   Note

   You should be cautious when granting ec2:TerminateInstances permissions because this action gives the recipient the ability to shut down any EC2 instance in the account, including those outside of Amazon EMR.

To create a VPC subnet to run Amazon EMR clusters

1. Sign in to the AWS Management Console and open the Amazon VPC console at https://console.aws.amazon.com/vpc/.

2. Create a VPC by clicking Get started creating a VPC. Make sure that the Region box is set to the same region where you'll be running your cluster. In this example, we're creating a VPC in the US East (N. Virginia) region.

   

3. Choose the VPC configuration by selecting one of the radio buttons.

   If the data used in the cluster is available on the Internet (e.g., Amazon S3 or Amazon RDS) select VPC with a Single Public Subnet Only.

   

   If the data used in the cluster is stored locally (e.g., an Oracle database), select VPC with Public and Private subnets and Hardware VPN Access.

   

4. Confirm the VPC settings. To work with Amazon EMR, the VPC must have both an Internet Gateway and a subnet.

   In a new VPC, the default DHCP configuration has a domain name by default (for example ec2.internal); therefore, you must click Edit VPC IP CIDR Block and enable the DNS Hostnames checkbox. For a VPC configured under EC2-Classic that has a domain name in the DHCP table, under DNS Settings, set the option Enable DNS hostname support for instances launched in this VPC or delete the domain-name in DHCP for the associated VPC. For more information, see Updating DNS Support for Your VPC.

   

5. A dialog box confirms that the Amazon VPC was successfully created. Click Close.

   

To find the VPC subnet identifier

• Click on Subnets in the navigation menu of the Amazon VPC console. The right pane displays information about the VPC, including its subnet identifier.

  

To launch a cluster on a VPC using the Amazon EMR console

1. In the Amazon EMR console, click Create New Job Flow.

   

2. Follow the instructions in the Create a New Job Flow wizard, selecting options that match the cluster you want to launch.

3. When you reach the ADVANCED OPTIONS page, choose the VPC subnet you created previously from the Amazon VPC Subnet Id box. If you have not created a subnet, click Create a VPC underneath the drop-down box to open the Amazon VPC console and create a VPC and subnet.

   

4. Continue the Create a Job Flow wizard until it is complete and the cluster is launched within the subnet specified in Step 3.

To launch a cluster on a VPC using the CLI

• After your VPC is configured, you can launch Amazon EMR clusters on it by using the --subnet argument and specifying the subnet address. This is illustrated in the following example, which creates a long-running cluster on the specified VPC subnet. In the directory where you installed the Amazon EMR CLI, run the following from the command line. For more information, see the Command Line Interface Reference for Amazon EMR.

  • Linux, UNIX, and Mac OS X users:

    ./elastic-mapreduce --create --alive --subnet subnet-identifier

  • Windows users:

    ruby elastic-mapreduce --create --alive --subnet subnet-identifier

To launch a cluster on a VPC using the API

• After your VPC is configured, you can launch Amazon EMR clusters on it by providing the VPC subnet identifier as the value for Ec2SubnetId, an optional String parameter on the JobFlowInstancesConfig structure.

  https://elasticmapreduce.amazonaws.com?
              Operation=RunJobFlow&
              Name=MyJobFlowName&
              LogUri=s3n%3A%2F%2Fmybucket%2Fsubdir&
              Instances.MasterInstanceType=m1.small&
              Instances.SlaveInstanceType=m1.small&
              Instances.InstanceCount=4&
              Instances.Ec2KeyName=myec2keyname&
              Instances.Placement.AvailabilityZone=us-east-1a&
              Instances.KeepJobFlowAliveWhenNoSteps=true&
              Instances.Ec2SubnetId=subnet-identifier&
              Steps.member.1.Name=MyStepName&
              Steps.member.1.ActionOnFailure=CONTINUE&
              Steps.member.1.HadoopJarStep.Jar=MyJarFile&
              Steps.member.1.HadoopJarStep.MainClass=MyMailClass&
              Steps.member.1.HadoopJarStep.Args.member.1=arg1&
              Steps.member.1.HadoopJarStep.Args.member.2=arg2&
              AWSAccessKeyId=AWS Access Key ID&
              SignatureVersion=2&
              SignatureMethod=HmacSHA256&
              Timestamp=2009-01-28T21%3A48%3A32.000Z&
              Signature=calculated value