Amazon Elastic MapReduce
Developer Guide (API Version 2009-03-31)
Did this page help you?  Yes | No |  Tell us about it...
« PreviousNext »
View the PDF for this guide.Go to the AWS Discussion Forum for this product.Go to the Kindle Store to download this guide in Kindle format.

Launch Spot Instances in a Cluster

When you launch a new instance group you can launch the Amazon EC2 instances in that group either as on-demand or as Spot Instances. The procedure for launching Spot Instances is the same as launching on-demand instances, except that you specify the bid price.

Launch a Cluster with Spot Instances Using the Amazon EMR Console

To launch an entire cluster with Spot Instances

  1. Open the Amazon EMR console at https://console.aws.amazon.com/elasticmapreduce/.

  2. Choose Create cluster.

  3. In the Hardware Configuration section, to run the master node as a Spot Instance, click Request Spot in the Master row and enter the maximum hourly rate you are willing to pay per instance in the Bid Price field. You can see the Spot Price for all of the current region's Availability Zones in the console by mousing over the information tooltip next to the Bid Price field. In most cases, you will want to enter a price higher than the current Spot Price.

  4. To run the core nodes as Spot Instances, click the Request Spot check box in the Core row and enter the maximum hourly rate you are willing to pay per instance in the Spot Bid Price field.

  5. To run the task nodes as Spot Instances, click the Request Spot check box in the Task row and enter the maximum hourly rate you are willing to pay per instance in the Spot Bid Price field.

    Note

    You can also create additional task instance groups (up to 48) using Spot Instances by clicking Add task instance group.

  6. Proceed to create the cluster as described in Plan an Amazon EMR Cluster.

Launch a Cluster with Spot Instances Using the AWS CLI

You can launch an entire cluster on Spot Instances using the AWS CLI, or you can launch specific instance groups on Spot Instances. With the CLI and API, you can choose the Availability Zone. If you do not select the Availability Zone, Amazon EMR will select it for you.

To launch an entire cluster with Spot Instances using the AWS CLI

To specify that an instance group should be launched with Spot Instances in the AWS CLI, type the create-cluster subcommand and use the --instance-groups parameter with the BidPrice argument.

  • To create a cluster where the master, core, and task instance groups all use Spot Instances, type the following command and replace myKey with the name of your EC2 key pair.

  • Linux, UNIX, and Mac OS X users:

    aws emr create-cluster --name "Spot cluster" --ami-version 3.3 --applications Name=Hive Name=Pig \
    --use-default-roles --ec2-attributes KeyName=myKey \
    --instance-groups InstanceGroupType=MASTER,InstanceType=m3.xlarge,InstanceCount=1,BidPrice=0.25 \
    InstanceGroupType=CORE,BidPrice=0.03,InstanceType=m3.xlarge,InstanceCount=2 \ 
    InstanceGroupType=TASK,BidPrice=0.10,InstanceType=m3.xlarge,InstanceCount=3 
  • Windows users:

    aws emr create-cluster --name "Spot cluster" --ami-version 3.3 --applications Name=Hive Name=Pig --use-default-roles --ec2-attributes KeyName=myKey --instance-groups InstanceGroupType=MASTER,InstanceType=m3.xlarge,InstanceCount=1,BidPrice=0.25 InstanceGroupType=CORE,BidPrice=0.03,InstanceType=m3.xlarge,InstanceCount=2 InstanceGroupType=TASK,BidPrice=0.10,InstanceType=m3.xlarge,InstanceCount=3

Note

If you have not previously created the default EMR service role and EC2 instance profile, type aws emr create-default-roles to create them before typing the create-cluster subcommand.

For more information on using Amazon EMR commands in the AWS CLI, see http://docs.aws.amazon.com/cli/latest/reference/emr.

Launch a Cluster with Spot Instances Using the Java SDK

To launch an entire cluster with Spot Instances using the Java SDK

  • To specify that an instance group should be launched as Spot Instances, set the withBidPrice and withMarket properties on the InstanceGroupConfig object that you instantiate for the instance group. The following code shows how to define master, core, and task instance groups that run as Spot Instances.

    InstanceGroupConfig instanceGroupConfigMaster = new InstanceGroupConfig()
    	.withInstanceCount(1)
    	.withInstanceRole(“MASTER”)
    	.withInstanceType(“m3.xlarge”)
    	.withMarket("SPOT")
    	.withBidPrice(“0.25”); 
    	
    InstanceGroupConfig instanceGroupConfigCore = new InstanceGroupConfig()
    	.withInstanceCount(4)
    	.withInstanceRole(“CORE”)
    	.withInstanceType(“m3.xlarge”)
    	.withMarket("SPOT")
    	.withBidPrice(“0.03”);
    	
    InstanceGroupConfig instanceGroupConfigTask = new InstanceGroupConfig()
    	.withInstanceCount(2)
    	.withInstanceRole(“TASK”)
    	.withInstanceType(“m3.xlarge”)
    	.withMarket("SPOT")
    	.withBidPrice(“0.10”);
    					

Launch Task Groups on Spot Instances Using the Amazon EMR Console

To launch task instance groups on Spot Instances

  1. Open the Amazon EMR console at https://console.aws.amazon.com/elasticmapreduce/.

  2. Choose Create cluster.

  3. In the Hardware Configuration section, to run only the task nodes as Spot Instances, click Request Spot in the Task row and enter the maximum hourly rate you are willing to pay per instance in the Bid Price field. You can see the Spot Price for all of the current region's Availability Zones in the console by mousing over the information tooltip next to the Bid Price field. In most cases, you will want to enter a price higher than the current Spot Price.

  4. To add additional task instance groups that use Spot Instances, click Add task instance group and follow the previous steps. You can add up to 48 task instance groups using the console.

  5. Proceed to create the cluster as described in Plan an Amazon EMR Cluster.

Launch Task Groups on Spot Instances Using the AWS CLI

Rather than launching all instance groups using Spot Instances, a more typical use case is to launch one or more task groups using Spot Instances. The following sections demonstrate launching task groups with Spot Instances using the console, the AWS CLI, the SDK, and the API.

To launch task instance groups on Spot Instances using the AWS CLI

To launch task instance groups on Spot Instances using the AWS CLI, specify the BidPrice for task instance groups.

  • To launch a single task instance group on Spot Instances, type the following command and replace myKey with the name of your EC2 key pair.

  • Linux, UNIX, and Mac OS X users:

    aws emr create-cluster --name "Spot cluster" --ami-version 3.3 --applications Name=Hive Name=Pig \
    --use-default-roles --ec2-attributes KeyName=myKey \
    --instance-groups InstanceGroupType=MASTER,InstanceType=m3.xlarge,InstanceCount=1 \
    InstanceGroupType=CORE,InstanceType=m3.xlarge,InstanceCount=2 \
    InstanceGroupType=TASK,BidPrice=0.03,InstanceType=m3.xlarge,InstanceCount=4
  • Windows users:

    aws emr create-cluster --name "Spot cluster" --ami-version 3.3 --applications Name=Hive Name=Pig --use-default-roles --ec2-attributes KeyName=myKey --instance-groups InstanceGroupType=MASTER,InstanceType=m3.xlarge,InstanceCount=1 InstanceGroupType=CORE,InstanceType=m3.xlarge,InstanceCount=2 InstanceGroupType=TASK,BidPrice=0.03,InstanceType=m3.xlarge,InstanceCount=4

Note

If you have not previously created the default EMR service role and EC2 instance profile, type aws emr create-default-roles to create them before typing the create-cluster subcommand.

  • To launch multiple task instance groups on Spot Instances, type the following command and replace myKey with the name of your EC2 key pair. You can add up to 5 task instance groups in a single command.

  • Linux, UNIX, and Mac OS X users:

    aws emr create-cluster --name "Spot cluster" --ami-version 3.3 --applications Name=Hive Name=Pig \
    --use-default-roles --ec2-attributes KeyName=myKey \
    --instance-groups InstanceGroupType=MASTER,InstanceType=m3.xlarge,InstanceCount=1 \
    InstanceGroupType=CORE,InstanceType=m3.xlarge,InstanceCount=2 \
    InstanceGroupType=TASK,BidPrice=0.03,InstanceType=m3.xlarge,InstanceCount=4 \
    InstanceGroupType=TASK,BidPrice=0.04,InstanceType=m3.xlarge,InstanceCount=2
  • Windows users:

    aws emr create-cluster --name "Spot cluster" --ami-version 3.3 --applications Name=Hive Name=Pig --use-default-roles --ec2-attributes KeyName=myKey --instance-groups InstanceGroupType=MASTER,InstanceType=m3.xlarge,InstanceCount=1 InstanceGroupType=CORE,InstanceType=m3.xlarge,InstanceCount=2 InstanceGroupType=TASK,BidPrice=0.03,InstanceType=m3.xlarge,InstanceCount=4 InstanceGroupType=TASK,BidPrice=0.04,InstanceType=m3.xlarge,InstanceCount=2

For more information on using Amazon EMR commands in the AWS CLI, see http://docs.aws.amazon.com/cli/latest/reference/emr.

Launch the Task Group on Spot Instances Using the Java SDK

To launch the task instance group on Spot Instances using the Java SDK

  • To specify that an instance group should be launched as Spot Instances, set the withBidPrice and withMarket properties on the InstanceGroupConfig object that you instantiate for the instance group. The following code creates a task instance group of type m1.large with an instance count of 10. It specifies $0.35 as the maximum bid price, which launches the task group on Spot Instances.

     InstanceGroupConfig instanceGroupConfigMaster = new InstanceGroupConfig()
    	.withInstanceCount(1)
    	.withInstanceRole(“MASTER”)
    	.withInstanceType(“m3.xlarge”)
    	
    InstanceGroupConfig instanceGroupConfigCore = new InstanceGroupConfig()
    	.withInstanceCount(4)
    	.withInstanceRole(“CORE”)
    	.withInstanceType(“m3.xlarge”)
    	
    InstanceGroupConfig instanceGroupConfig = new InstanceGroupConfig()
    	.withInstanceCount(10)
    	.withInstanceRole(“TASK”)
    	.withInstanceType(“m3.xlarge”)
    	.withMarket"("SPOT")
    	.withBidPrice(“0.35”);