Amazon Elastic MapReduce
Developer Guide (API Version 2009-03-31)
« PreviousNext »
View the PDF for this guide.Go to the AWS Discussion Forum for this product.Go to the Kindle Store to download this guide in Kindle format.Did this page help you?  Yes | No |  Tell us about it...

Launch Spot Instances in a Cluster

When you launch a new instance group you can launch the EC2 instances in that group either as on-demand or as Spot Instances. The procedure for launching Spot Instances is the same as launching on-demand instances, except that you specify additional information such as the market type and the bid price.

Amazon EMR Console

To launch an entire cluster on Spot Instances

Setting all nodes as Spot Instances
  1. Sign in to the AWS Management Console and open the Amazon Elastic MapReduce console at https://console.aws.amazon.com/elasticmapreduce/vnext/.

  2. Click Create Cluster.

    Create Cluster
  3. In the Hardware Configuration section, to run the master node as a Spot Instance, select Request Spot in the Master row and enter the maximum hourly rate you are willing to pay per instance in the Bid price field that appears. You can look up the current Spot Price for instances on the Amazon EC2 Pricing page. In most cases, you will want to enter a price higher than the current Spot Price.

  4. To run the core nodes as Spot Instances, select the Request Spot Instance check box in the Core row and enter the maximum hourly rate you are willing to pay per instance in the Spot Bid Price text box that appears.

  5. To run the task nodes as Spot Instances, select the Request Spot Instance check box in the Task row and enter the maximum hourly rate you are willing to pay per instance in the Spot Bid Price text box that appears.

  6. Proceed to create the cluster as described in Plan an Amazon EMR Cluster.

CLI

To launch an entire cluster on Spot Instances

  • To specify that an instance group should be launched as Spot Instances, use the --bid-price parameter. The following example shows how to create a cluster where the master, core, and task instance groups are all running as Spot Instances. The following code launches a cluster only after until the requests for the master and core instances have been completely fulfilled.

    In the directory where you installed the Amazon EMR CLI, run the following from the command line. For more information, see the Command Line Interface Reference for Amazon EMR.

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --create --alive --name “Spot Cluster” \
      --instance-group master --instance-type m1.large --instance-count 1 --bid-price 0.25 \
      --instance-group core --instance-type m1.small --instance-count 4 --bid-price 0.03 \
      --instance-group task --instance-type c1.medium --instance-count 2 --bid-price 0.10
    • Windows users:

      ruby elastic-mapreduce --create --alive --name “Spot Cluster” --instance-group master --instance-type m1.large --instance-count 1 --bid-price 0.25 --instance-group core --instance-type m1.small --instance-count 4 --bid-price 0.03 --instance-group task --instance-type c1.medium --instance-count 2 --bid-price 0.10

Java SDK

To launch an entire cluster on Spot Instances

  • To specify that an instance group should be launched as Spot Instances, set the withBidPrice and withMarket properties on the InstanceGroupConfig object that you instantiate for the instance group. The following code shows how to define master, core, and task instance groups that run as Spot Instances.

    InstanceGroupConfig instanceGroupConfigMaster = new InstanceGroupConfig()
    	.withInstanceCount(1)
    	.withInstanceRole(“MASTER”)
    	.withInstanceType(“m1.large”)
    	.withMarket("SPOT")
    	.withBidPrice(“0.25”); 
    	
    InstanceGroupConfig instanceGroupConfigCore = new InstanceGroupConfig()
    	.withInstanceCount(4)
    	.withInstanceRole(“CORE”)
    	.withInstanceType(“m1.small”)
    	.withMarket("SPOT")
    	.withBidPrice(“0.03”);
    	
    InstanceGroupConfig instanceGroupConfigTask = new InstanceGroupConfig()
    	.withInstanceCount(2)
    	.withInstanceRole(“TASK”)
    	.withInstanceType(“c1.medium”)
    	.withMarket("SPOT")
    	.withBidPrice(“0.10”);
    					

API

To launch an entire cluster on Spot Instances

  • To specify that an instance group should be launched as Spot Instances, set the BidPrice and Market properties of the InstanceGroupDetail members of the InstanceGroupDetailList. The code below shows how to define master, core, and task instance groups that run as Spot Instances.

    Example Sample Request

    https://elasticmapreduce.amazonaws.com?Operation=RunJobFlow
    &Name=MyJobFlowName 
    &LogUri=s3n%3A%2F%2Fmybucket%2Fsubdir
    &Instances.MasterInstanceType=m1.large 
    &Instances.SlaveInstanceType=m1.small
    &Instances.InstanceCount=4 
    &Instances.Ec2KeyName=myec2keyname
    &Instances.Placement.AvailabilityZone=us-east-1a
    &Instances.KeepJobFlowAliveWhenNoSteps=true 
    &Instances.TerminationProtected=true
    &Instances.InstanceGroups.member.1.InstanceRole=MASTER
    &Instances.InstanceGroups.member.1.Market=SPOT
    &Instances.InstanceGroups.member.1.BidPrice=.25
    &Instances.InstanceGroups.member.2.InstanceRole=CORE
    &Instances.InstanceGroups.member.2.Market=SPOT
    &Instances.InstanceGroups.member.2.BidPrice=.03
    &Instances.InstanceGroups.member.3.InstanceRole=TASK
    &Instances.InstanceGroups.member.3.Market=SPOT
    &Instances.InstanceGroups.member.3.BidPrice=.03
    &Steps.member.1.Name=MyStepName
    &Steps.member.1.ActionOnFailure=CONTINUE
    &Steps.member.1.HadoopJarStep.Jar=MyJarFile
    &Steps.member.1.HadoopJarStep.MainClass=MyMainClass
    &Steps.member.1.HadoopJarStep.Args.member.1=arg1
    &Steps.member.1.HadoopJarStep.Args.member.2=arg2 
    &AuthParams
    					

    Example Sample Response

    <RunJobFlowResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31">
    	<RunJobFlowResult>
    		<JobFlowId>
    			j-3UN6WX5RRO2AG 
    		</JobFlowId>
    	</RunJobFlowResult>
    	<ResponseMetadata>
    		<RequestId>
    			8296d8b8-ed85-11dd-9877-6fad448a8419 
    		</RequestId> 
    	</ResponseMetadata>
    </RunJobFlowResponse>						
    					

Amazon EMR Console

To launch only the task instance group on Spot Instances

Setting the task nodes as Spot Instances
  1. Sign in to the AWS Management Console and open the Amazon Elastic MapReduce console at https://console.aws.amazon.com/elasticmapreduce/vnext/.

  2. Click Create Cluster.

    Create Cluster
  3. In the Hardware Configuration section, to run only the task nodes as Spot Instances, select Request Spot Instance in the Task row and enter the maximum hourly rate you are willing to pay per instance in the Bid price field that appears. You can look up the current Spot Price for instances on the Amazon EC2 Pricing page. In most cases, you will want to enter a price higher than the current Spot Price.

  4. Proceed to create the cluster as described in Plan an Amazon EMR Cluster.

CLI

To launch only the task instance group on Spot Instances

  • To specify that an instance group should be launched as Spot Instances, use the --bid-price parameter. The following example shows how to create a cluster where only the task instance group is running as Spot Instances. The code below will launch a cluster even if the request for Spot Instances can’t be fulfilled. In that case, Amazon EMR will add task nodes to the cluster if it is still running when the Spot Price falls below the bid price.

    In the directory where you installed the Amazon EMR CLI, run the following from the command line. For more information, see the Command Line Interface Reference for Amazon EMR.

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --create --alive --name “Spot Task Group” \
      --instance-group master --instance-type m1.large \
      --instance-count 1 \
      --instance-group core --instance-type m1.large \
      --instance-count 2 \
      --instance-group task --instance-type m1.small \
      --instance-count 4 --bid-price 0.03 
    • Windows users:

      ruby elastic-mapreduce --create --alive --name “Spot Task Group” --instance-group master --instance-type m1.large --instance-count 1 --instance-group core --instance-type m1.large --instance-count 2 --instance-group task --instance-type m1.small --instance-count 4 --bid-price 0.03 

Java SDK

To launch only the task instance group on Spot Instances

  • To specify that an instance group should be launched as Spot Instances, set the withBidPrice and withMarket properties on the InstanceGroupConfig object that you instantiate for the instance group. The following code creates a task instance group of type m1.large with an instance count of 10. It specifies $0.35 as the maximum bid price, and will run as Spot Instances.

     InstanceGroupConfig instanceGroupConfigMaster = new InstanceGroupConfig()
    	.withInstanceCount(1)
    	.withInstanceRole(“MASTER”)
    	.withInstanceType(“m1.large”)
    	
    InstanceGroupConfig instanceGroupConfigCore = new InstanceGroupConfig()
    	.withInstanceCount(4)
    	.withInstanceRole(“CORE”)
    	.withInstanceType(“m1.small”)
    	
    InstanceGroupConfig instanceGroupConfig = new InstanceGroupConfig()
    	.withInstanceCount(10)
    	.withInstanceRole(“TASK”)
    	.withInstanceType(“m1.large”)
    	.withMarket"("SPOT")
    	.withBidPrice(“0.35”);
                	

API

To launch only the task instance group on Spot Instances

  • To specify that an instance group should be launched as Spot Instances, set the BidPrice and Market properties of the TASK InstanceGroupDetail member of the InstanceGroupDetailList. The following code shows how to define only the task instance group to run as Spot Instances.

    The following example shows how to format the request.

    Example Sample Request

    https://elasticmapreduce.amazonaws.com?Operation=RunJobFlow
    &Name=MyJobFlowName 
    &LogUri=s3n%3A%2F%2Fmybucket%2Fsubdir
    &Instances.MasterInstanceType=m1.large 
    &Instances.SlaveInstanceType=m1.small
    &Instances.InstanceCount=4 
    &Instances.Ec2KeyName=myec2keyname
    &Instances.Placement.AvailabilityZone=us-east-1a
    &Instances.KeepJobFlowAliveWhenNoSteps=true 
    &Instances.TerminationProtected=true
     &Instances.InstanceGroups.member.1.InstanceRole=MASTER
    &Instances.InstanceGroups.member.2.InstanceRole=CORE
    &Instances.InstanceGroups.member.3.InstanceRole=TASK
    &Instances.InstanceGroups.member.3.Market=SPOT
     &Instances.InstanceGroups.member.3.BidPrice=.03
    &Steps.member.1.Name=MyStepName
    &Steps.member.1.ActionOnFailure=CONTINUE
    &Steps.member.1.HadoopJarStep.Jar=MyJarFile
    &Steps.member.1.HadoopJarStep.MainClass=MyMainClass
    &Steps.member.1.HadoopJarStep.Args.member.1=arg1
    &Steps.member.1.HadoopJarStep.Args.member.2=arg2 
    &AuthParams
    					

    Example Sample Response

    <RunJobFlowResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31">
       <RunJobFlowResult> 
          <JobFlowId>
             j-3UN6WX5RRO2AG 
          </JobFlowId>
       </RunJobFlowResult> 
       <ResponseMetadata> 
          <RequestId>
             8296d8b8-ed85-11dd-9877-6fad448a8419 
          </RequestId>
       </ResponseMetadata>
    </RunJobFlowResponse>