Amazon Elastic MapReduce
Developer Guide (API Version 2009-03-31)
Did this page help you?  Yes | No |  Tell us about it...
« PreviousNext »
View the PDF for this guide.Go to the AWS Discussion Forum for this product.Go to the Kindle Store to download this guide in Kindle format.

Launch Spot Instances in a Cluster

When you launch a new instance group you can launch the Amazon EC2 instances in that group either as on-demand or as Spot Instances. The procedure for launching Spot Instances is the same as launching on-demand instances, except that you specify the bid price.

Launch a Cluster with Spot Instances Using the Amazon EMR Console

To launch an entire cluster with Spot Instances

  1. Open the Amazon Elastic MapReduce console at https://console.aws.amazon.com/elasticmapreduce/.

  2. Click Create cluster.

  3. In the Hardware Configuration section, to run the master node as a Spot Instance, select Request Spot in the Master row and enter the maximum hourly rate you are willing to pay per instance in the Bid price field that appears. You can look up the current Spot Price for instances on the Amazon EC2 Pricing page. In most cases, you will want to enter a price higher than the current Spot Price.

  4. To run the core nodes as Spot Instances, select the Request Spot check box in the Core row and enter the maximum hourly rate you are willing to pay per instance in the Bid Price field.

  5. To run the task nodes as Spot Instances, select the Request spot check box in the Task row and enter the maximum hourly rate you are willing to pay per instance in the Bid Price field.

    Setting all nodes as Spot Instances
  6. Proceed to create the cluster as described in Plan an Amazon EMR Cluster.

Launch a Cluster with Spot Instances Using the AWS CLI or Amazon EMR CLI

You can launch an entire cluster on Spot Instances using the AWS CLI or Amazon EMR CLI, or you can launch specific instance groups on Spot Instances.

To launch an entire cluster with Spot Instances using the AWS CLI

To specify that an instance group should be launched with Spot Instances in the AWS CLI, type the create-cluster subcommand and use the --instance-groups parameter with the BidPrice argument.

  • To create a cluster where the master, core, and task instance groups all use Spot Instances, type the following command and replace myKey with the name of your EC2 key pair.

  • Linux, UNIX, and Mac OS X users:

    aws emr create-cluster --name "Spot cluster" --ami-version 3.3 --applications Name=Hue Name=Hive Name=Pig \
    --use-default-roles --ec2-attributes KeyName=myKey \
    --instance-groups InstanceGroupType=MASTER,InstanceType=m3.xlarge,InstanceCount=1,BidPrice=0.25 \
    InstanceGroupType=CORE,BidPrice=0.03,InstanceType=m3.xlarge,InstanceCount=2 \ 
    InstanceGroupType=TASK,BidPrice=0.10,InstanceType=m3.xlarge,InstanceCount=3 
  • Windows users:

    aws emr create-cluster --name "Spot cluster" --ami-version 3.3 --applications Name=Hue Name=Hive Name=Pig --use-default-roles --ec2-attributes KeyName=myKey --instance-groups InstanceGroupType=MASTER,InstanceType=m3.xlarge,InstanceCount=1,BidPrice=0.25 InstanceGroupType=CORE,BidPrice=0.03,InstanceType=m3.xlarge,InstanceCount=2 InstanceGroupType=TASK,BidPrice=0.10,InstanceType=m3.xlarge,InstanceCount=3

Note

If you have not previously created the default EMR service role and EC2 instance profile, type aws emr create-default-roles to create them before typing the create-cluster subcommand.

For more information on using Amazon EMR commands in the AWS CLI, see http://docs.aws.amazon.com/cli/latest/reference/emr.

To launch an entire cluster with Spot Instances using the Amazon EMR CLI

Note

The Amazon EMR CLI is no longer under feature development. Customers are encouraged to use the Amazon EMR commands in the AWS CLI instead.

  • To specify that an instance group should be launched as Spot Instances, use the --bid-price parameter. The following example shows how to create a cluster where the master, core, and task instance groups are all running as Spot Instances. The following code launches a cluster only after until the requests for the master and core instances have been completely fulfilled. In the directory where you installed the Amazon EMR CLI, run the following from the command line. For more information, see the Command Line Interface Reference for Amazon EMR.

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --create --alive --name "Spot Cluster" \
      --instance-group master --instance-type m1.large --instance-count 1 --bid-price 0.25 \
      --instance-group core --instance-type m1.large --instance-count 4 --bid-price 0.03 \
      --instance-group task --instance-type c1.medium --instance-count 2 --bid-price 0.10
    • Windows users:

      ruby elastic-mapreduce --create --alive --name "Spot Cluster" --instance-group master --instance-type m1.large --instance-count 1 --bid-price 0.25 --instance-group core --instance-type m1.large --instance-count 4 --bid-price 0.03 --instance-group task --instance-type c1.medium --instance-count 2 --bid-price 0.10

Launch a Cluster with Spot Instances Using the Java SDK

To launch an entire cluster with Spot Instances using the Java SDK

  • To specify that an instance group should be launched as Spot Instances, set the withBidPrice and withMarket properties on the InstanceGroupConfig object that you instantiate for the instance group. The following code shows how to define master, core, and task instance groups that run as Spot Instances.

    InstanceGroupConfig instanceGroupConfigMaster = new InstanceGroupConfig()
    	.withInstanceCount(1)
    	.withInstanceRole(“MASTER”)
    	.withInstanceType(“m1.large”)
    	.withMarket("SPOT")
    	.withBidPrice(“0.25”); 
    	
    InstanceGroupConfig instanceGroupConfigCore = new InstanceGroupConfig()
    	.withInstanceCount(4)
    	.withInstanceRole(“CORE”)
    	.withInstanceType(“m1.large”)
    	.withMarket("SPOT")
    	.withBidPrice(“0.03”);
    	
    InstanceGroupConfig instanceGroupConfigTask = new InstanceGroupConfig()
    	.withInstanceCount(2)
    	.withInstanceRole(“TASK”)
    	.withInstanceType(“c1.medium”)
    	.withMarket("SPOT")
    	.withBidPrice(“0.10”);
    					

Launch a Cluster with Spot Instances Using the API

To launch an entire cluster with Spot Instances using the API

  • To specify that an instance group should be launched as Spot Instances, set the BidPrice and Market properties of the InstanceGroupDetail members of the InstanceGroupDetailList. The following code shows how to define master, core, and task instance groups that run as Spot Instances.

    Example Sample Request

    https://elasticmapreduce.amazonaws.com?Operation=RunJobFlow
    &Name=MyJobFlowName 
    &LogUri=s3n%3A%2F%2Fmybucket%2Fsubdir
    &Instances.MasterInstanceType=m1.large 
    &Instances.SlaveInstanceType=m1.large
    &Instances.InstanceCount=4 
    &Instances.Ec2KeyName=myec2keyname
    &Instances.Placement.AvailabilityZone=us-east-1a
    &Instances.KeepJobFlowAliveWhenNoSteps=true 
    &Instances.TerminationProtected=true
    &Instances.InstanceGroups.member.1.InstanceRole=MASTER
    &Instances.InstanceGroups.member.1.Market=SPOT
    &Instances.InstanceGroups.member.1.BidPrice=.25
    &Instances.InstanceGroups.member.2.InstanceRole=CORE
    &Instances.InstanceGroups.member.2.Market=SPOT
    &Instances.InstanceGroups.member.2.BidPrice=.03
    &Instances.InstanceGroups.member.3.InstanceRole=TASK
    &Instances.InstanceGroups.member.3.Market=SPOT
    &Instances.InstanceGroups.member.3.BidPrice=.03
    &Steps.member.1.Name=MyStepName
    &Steps.member.1.ActionOnFailure=CONTINUE
    &Steps.member.1.HadoopJarStep.Jar=MyJarFile
    &Steps.member.1.HadoopJarStep.MainClass=MyMainClass
    &Steps.member.1.HadoopJarStep.Args.member.1=arg1
    &Steps.member.1.HadoopJarStep.Args.member.2=arg2 
    &AuthParams
    					


    Example Sample Response

    <RunJobFlowResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31">
    	<RunJobFlowResult>
    		<JobFlowId>
    			j-3UNXXXXXXO2AG 
    		</JobFlowId>
    	</RunJobFlowResult>
    	<ResponseMetadata>
    		<RequestId>
    			8296d8b8-ed85-11dd-9877-6fad448a8419 
    		</RequestId> 
    	</ResponseMetadata>
    </RunJobFlowResponse>						
    					


Launch the Task Group on Spot Instances Using the Amazon EMR Console

To launch only the task instance group on Spot Instances

  1. Open the Amazon Elastic MapReduce console at https://console.aws.amazon.com/elasticmapreduce/.

  2. Click Create cluster.

  3. In the Hardware Configuration section, to run only the task nodes as Spot Instances, select Request spot in the Task row and enter the maximum hourly rate you are willing to pay per instance in the Bid price field. You can look up the current Spot Price for instances on the Amazon EC2 Pricing page. In most cases, you will want to enter a price higher than the current Spot Price.

    Setting the task nodes as Spot Instances
  4. Proceed to create the cluster as described in Plan an Amazon EMR Cluster.

Launch the Task Group on Spot Instances Using the AWS CLI or Amazon EMR CLI

Rather than launching all instance groups using Spot Instances, a more typical use case is to launch the task group using Spot Instances. The following sections demonstrate launching the task group with Spot Instances using the console, the AWS CLI, the Amazon EMR CLI, the SDK, and the API.

To launch the task instance group on Spot Instances using the AWS CLI

To launch the task instance group on Spot Instances using the AWS CLI, specify the BidPrice for the task instance group.

  • To launch the task instance group on Spot Instances, type the following command and replace myKey with the name of your EC2 key pair.

  • Linux, UNIX, and Mac OS X users:

    aws emr create-cluster --name "Spot cluster" --ami-version 3.3 --applications Name=Hue Name=Hive Name=Pig \
    --use-default-roles --ec2-attributes KeyName=myKey \
    --instance-groups InstanceGroupType=MASTER,InstanceType=m3.xlarge,InstanceCount=1 \
    InstanceGroupType=CORE,InstanceType=m3.xlarge,InstanceCount=2 \
    InstanceGroupType=TASK,BidPrice=0.03,InstanceType=m3.xlarge,InstanceCount=4
  • Windows users:

    aws emr create-cluster --name "Spot cluster" --ami-version 3.3 --applications Name=Hue Name=Hive Name=Pig --use-default-roles --ec2-attributes KeyName=myKey --instance-groups InstanceGroupType=MASTER,InstanceType=m3.xlarge,InstanceCount=1 InstanceGroupType=CORE,InstanceType=m3.xlarge,InstanceCount=2 InstanceGroupType=TASK,BidPrice=0.03,InstanceType=m3.xlarge,InstanceCount=4

Note

If you have not previously created the default EMR service role and EC2 instance profile, type aws emr create-default-roles to create them before typing the create-cluster subcommand.

For more information on using Amazon EMR commands in the AWS CLI, see http://docs.aws.amazon.com/cli/latest/reference/emr.

To launch the task instance group on Spot Instances using the Amazon EMR CLI

Note

The Amazon EMR CLI is no longer under feature development. Customers are encouraged to use the Amazon EMR commands in the AWS CLI instead.

  • To specify that an instance group should be launched as Spot Instances, use the --bid-price parameter. The following example shows how to create a cluster where only the task instance group uses Spot Instances. The command launches a cluster even if the request for Spot Instances cannot be fulfilled. In that case, Amazon EMR adds task nodes to the cluster if it is still running when the Spot Price falls below the bid price. In the directory where you installed the Amazon EMR CLI, type the following command. For more information, see the Command Line Interface Reference for Amazon EMR.

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --create --alive --name "Spot Task Group" \
      --instance-group master --instance-type m1.large \
      --instance-count 1 \
      --instance-group core --instance-type m1.large \
      --instance-count 2 \
      --instance-group task --instance-type m1.large \
      --instance-count 4 --bid-price 0.03 
    • Windows users:

      ruby elastic-mapreduce --create --alive --name "Spot Task Group" --instance-group master --instance-type m1.large --instance-count 1 --instance-group core --instance-type m1.large --instance-count 2 --instance-group task --instance-type m1.small --instance-count 4 --bid-price 0.03 

Launch the Task Group on Spot Instances Using the Java SDK

To launch the task instance group on Spot Instances using the Java SDK

  • To specify that an instance group should be launched as Spot Instances, set the withBidPrice and withMarket properties on the InstanceGroupConfig object that you instantiate for the instance group. The following code creates a task instance group of type m1.large with an instance count of 10. It specifies $0.35 as the maximum bid price, which launches the task group on Spot Instances.

     InstanceGroupConfig instanceGroupConfigMaster = new InstanceGroupConfig()
    	.withInstanceCount(1)
    	.withInstanceRole(“MASTER”)
    	.withInstanceType(“m1.large”)
    	
    InstanceGroupConfig instanceGroupConfigCore = new InstanceGroupConfig()
    	.withInstanceCount(4)
    	.withInstanceRole(“CORE”)
    	.withInstanceType(“m1.large”)
    	
    InstanceGroupConfig instanceGroupConfig = new InstanceGroupConfig()
    	.withInstanceCount(10)
    	.withInstanceRole(“TASK”)
    	.withInstanceType(“m1.large”)
    	.withMarket"("SPOT")
    	.withBidPrice(“0.35”);
                	

Launch the Task Group on Spot Instances Using the API

To launch the task instance group on Spot Instances using the API

  • To specify that an instance group should be launched as Spot Instances, set the BidPrice and Market properties of the TASK InstanceGroupDetail member of the InstanceGroupDetailList. The following code shows how to format the request to define the task instance group using Spot Instances.

    Example Sample Request

    https://elasticmapreduce.amazonaws.com?Operation=RunJobFlow
    &Name=MyJobFlowName 
    &LogUri=s3n%3A%2F%2Fmybucket%2Fsubdir
    &Instances.MasterInstanceType=m1.large 
    &Instances.SlaveInstanceType=m1.large
    &Instances.InstanceCount=4 
    &Instances.Ec2KeyName=myec2keyname
    &Instances.Placement.AvailabilityZone=us-east-1a
    &Instances.KeepJobFlowAliveWhenNoSteps=true 
    &Instances.TerminationProtected=true
     &Instances.InstanceGroups.member.1.InstanceRole=MASTER
    &Instances.InstanceGroups.member.2.InstanceRole=CORE
    &Instances.InstanceGroups.member.3.InstanceRole=TASK
    &Instances.InstanceGroups.member.3.Market=SPOT
     &Instances.InstanceGroups.member.3.BidPrice=.03
    &Steps.member.1.Name=MyStepName
    &Steps.member.1.ActionOnFailure=CONTINUE
    &Steps.member.1.HadoopJarStep.Jar=MyJarFile
    &Steps.member.1.HadoopJarStep.MainClass=MyMainClass
    &Steps.member.1.HadoopJarStep.Args.member.1=arg1
    &Steps.member.1.HadoopJarStep.Args.member.2=arg2 
    &AuthParams
    					 


    Example Sample Response

    <RunJobFlowResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31">
       <RunJobFlowResult> 
          <JobFlowId>
             j-3UNXXXXXXO2AG 
          </JobFlowId>
       </RunJobFlowResult> 
       <ResponseMetadata> 
          <RequestId>
             8296d8b8-ed85-11dd-9877-6fad448a8419 
          </RequestId>
       </ResponseMetadata>
    </RunJobFlowResponse>