Amazon Elastic MapReduce
Developer Guide (API Version 2009-03-31)
« PreviousNext »
View the PDF for this guide.Go to the AWS Discussion Forum for this product.Go to the Kindle Store to download this guide in Kindle format.Did this page help you?  Yes | No |  Tell us about it...

Change the Number of Spot Instances in a Cluster

With some restrictions, you can modify the number of Spot Instances in a cluster. For example, you cannot change the number of instances in the master instance group, it always has one instance. To prevent data loss, you can add, but not remove core nodes from an instance group. Task nodes do not store state, and so they can be added or removed from a running cluster.

You can only define an instance group as running as Spot Instances when it is created, so for example, if you launched the core instance group as on-demand, you would not be able to change its market type to Spot Instances later.

Note

It’s possible to automatically modify the number of slave nodes in a cluster between cluster steps. You just include a predefined step in your workflow that changes the number of requested Spot Instances.

In the directory where you installed the Amazon EMR CLI, run the following from the command line. For more information, see the Command Line Interface Reference for Amazon EMR.

  • Linux, UNIX, and Mac OS X users:

    ./elastic-mapreduce --jobflow JobFlowId \
    --jar s3://elasticmapreduce/libs/resize-job-flow/0.1/resize-job-flow.jar \
    --args --modify-instance-group,task,--instance-count,4	
  • Windows users:

    ruby elastic-mapreduce --jobflow JobFlowId --jar s3://elasticmapreduce/libs/resize-job-flow/0.1/resize-job-flow.jar --args --modify-instance-group,task,--instance-count,4

The following examples show how to add task and core instance groups to a running cluster by increasing the number of instances in an instance group. For task nodes, you can also decrease the number of instances. The procedure is the same as shown in the examples that follow, but instead of specifying a larger number of instances than is currently running, you would specify a smaller number.

Note

If you are running a cluster that contains only a master node, you cannot add instance groups to that cluster. A cluster must have one or more core instances for you to be able to add or modify instance groups.

Note

Adding nodes to a running cluster is not currently supported in the Amazon EMR console.

CLI

To add Spot Instances to a running cluster

  • You can use --add-instance-group to add a task instance group of Spot Instances to a running cluster.

    In the directory where you installed the Amazon EMR CLI, run the following from the command line. For more information, see the Command Line Interface Reference for Amazon EMR.

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --jobflow JobFlowId \
      --add-instance-group task  --instance-type m1.small \
      --instance-count 5 --bid-price 0.05
    • Windows users:

      ruby elastic-mapreduce --jobflow JobFlowId --add-instance-group task  --instance-type m1.small --instance-count 5 --bid-price 0.05

CLI

To increase or decrease the number of Spot Instances in a running cluster

  • You can change the number of requested Spot Instances in a running cluster by calling the --modify-instance-group and --instance-count on the command line. Note that you can only increase the number of core instances in your cluster while you can increase or decrease the number of task instances. Setting the number of task instances to zero will remove all Spot Instances.

    In the directory where you installed the Amazon EMR CLI, run the following from the command line. For more information, see the Command Line Interface Reference for Amazon EMR.

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --jobflow JobFlowId \
      --modify-instance-group task --instance-count 5
    • Windows users:

      ruby elastic-mapreduce --jobflow JobFlowId --modify-instance-group task --instance-count 5

    This will change the number of requested TASK InstanceGroup instances to 5.

Java SDK

To add Spot Instances to a running cluster

  • To specify that an instance group should be launched as Spot Instances, set the withBidPrice and withMarket properties on the InstanceGroupConfig object that you instantiate for the instance group. The following code creates a task instance group of type m1.large with an instance count of 10. It specifies $0.35 as the maximum bid price, and will run as Spot Instances.

    When you make the call to modify the instance group, pass this object instance in.

    InstanceGroupConfig instanceGroupConfig = new InstanceGroupConfig()
    	.withInstanceCount(10)
    	.withInstanceRole(“TASK”)
    	.withInstanceType(“m1.large”)
    	.withMarket("SPOT")
    	.withBidPrice(“0.35”);
    					

API

To add Spot Instances to a running cluster

  • The following sample request increases the number of task nodes in the task instance group to eight and requests that they be launched as Spot Instances with an hourly bid price of .35.

    The following is an example of the request you would send in to Amazon EMR.

    Sample Request

    https://elasticmapreduce.amazonaws.com?Operation=AddInstanceGroups
    &InstanceGroups.member.1.InstanceGroupId=i-3UN6WX5RRO2AG 
    &InstanceGroups.member.1.InstanceRequestCount=8
    &InstanceGroups.member.1.InstanceRole=TASK
    &InstanceGroups.member.1.Market=SPOT
    &InstanceGroups.member.1.BidPrice=.35
    &AuthParams
    

    Sample Response

    <ModifyInstanceGroupsResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31">
       <ResponseMetadata>
          <RequestId>
             2690d7eb-ed86-11dd-9877-6fad448a8419
          </RequestId>
       </ResponseMetadata>
    </ModifyInstanceGroupsResponse>