Amazon Elastic MapReduce
Developer Guide (API Version 2009-03-31)
Did this page help you?  Yes | No |  Tell us about it...
« PreviousNext »
View the PDF for this guide.Go to the AWS Discussion Forum for this product.Go to the Kindle Store to download this guide in Kindle format.

Change the Number of Spot Instances in a Cluster

With some restrictions, you can modify the number of Spot Instances in a cluster. For example, because the master instance group contains only one instance, you cannot modify the master group's instance count whether or not the instance group is using a Spot Instance.

If you are running a cluster that contains only a master node, you cannot add instance groups or instances to that cluster. A cluster must have one or more core instances for you to be able to add or modify instance groups.

You can only define an instance group using Spot Instances when it is created. For example, if you launch the core instance group using on-demand instances, you cannot change them to Spot Instances later.

The following examples show how to change the number of Spot Instances in an instance group using the console, CLI, SDK, and API.

Change the Number of Spot Instances Using the Amazon EMR Console

If you launch a cluster using Spot Instances for one or more instance groups, you can change the number of requested Spot Instances for those groups using the Amazon EMR console. Note that you can only increase the number of core instances in your cluster while you can increase or decrease the number of task instances. Setting the number of task instances to zero removes all Spot Instances for that instance group.

To change the number of Spot Instances in an instance group using the console

  1. In the Amazon EMR console, on the Cluster List page, click the link for your cluster.

  2. Click Resize.

  3. In the Resize Cluster dialog, in the Count column, click the Resize link for the instance groups to which you wish to add Spot Instances.

  4. To add core or task instances, type a higher number in the Count column and click the check mark to confirm the change. To reduce the number of task instances, type a lower number in the Count column and click the check mark to confirm the change. After changing the instance count, click Close.

    Resizing spot instance groups in the console

    Note

    You cannot change the bid price when adding or removing Spot Instances from instance groups using the console. Also note, changing the instance count of the task group to 0 removes all Spot Instances but not the instance group.

Change the Number of Spot Instances Using the AWS CLI or Amazon EMR CLI

Using the AWS CLI, you can: increase the number of Spot Instances in the core instance group, increase or decrease the number of Spot Instances in the task instance group, add a task instance group containing Spot Instances, or terminate a Spot Instance in the core instance group.

Using the Amazon EMR CLI, you can: increase the number of Spot Instances in the core instance group, increase or decrease the number of Spot Instances in the task instance group, or add a task instance group containing Spot Instances.

To change the number of Spot Instances for instance groups using the AWS CLI

You can add Spot Instances to the core group or task group, and you can remove instances from the task group using the AWS CLI modify-instance-groups subcommand with the InstanceCount parameter. To add instances to the core or task groups, increase the InstanceCount. To reduce the number of instances in the task group, decrease the InstanceCount. Changing the instance count of the task group to 0 removes all Spot Instances but not the instance group.

  1. Retrieve the cluster ID for the cluster you wish to alter using the console or by typing:

    aws emr list-clusters

  2. To change the instance count for an instance group, you need the InstanceGroupId. To retrieve the InstanceGroupId, use the describe-cluster subcommand with the cluster ID:

    aws emr describe-cluster --cluster-id string

    The output is a JSON object called Cluster that contains the ID of each instance group.

    You can also pipe the JSON object to a program like grep or jq. To retrieve a cluster's instance group name and ID using jq on Unix-based systems, type the following:

    aws emr describe-cluster --cluster-id string | jq ".Cluster.InstanceGroups[] | {name,Id}"
  3. To increase the number of Spot Instances in the task instance group from 3 to 4, type:

    aws emr modify-instance-groups --instance-groups InstanceGroupId=ig-31JXXXXXXBTO,InstanceCount=4

For more information on using Amazon EMR commands in the AWS CLI, see http://docs.aws.amazon.com/cli/latest/reference/emr.

To add a task instance group with Spot Instances to a cluster using the AWS CLI

Using the AWS CLI, you can add a task instance group of Spot Instances to a cluster with the add-instance-groups subcommand.

  • To add a task instance group of Spot Instances to a cluster, type:

    aws emr add-instance-groups --cluster-id j-JXBXXXXXX37R --instance-groups InstanceCount=6,BidPrice=.25,InstanceGroupType=task,InstanceType=m3.xlarge

    Note

    You cannot add a task instance group to a cluster if the cluster was created with a task group, even if the instance count for the group is 0.

    For more information on using Amazon EMR commands in the AWS CLI, see http://docs.aws.amazon.com/cli/latest/reference/emr.

To terminate a Spot Instance in the core instance group using the AWS CLI

Using the AWS CLI, you can terminate a Spot Instance in the core instance group with the modify-instance-groups subcommand. This should be done with caution. Terminating an instance in the core instance group risks data loss, and the instance is not automatically replaced. To terminate a specific instance you need the instance group ID (returned by the aws emr describe-cluster --cluster-id command) and the instance ID (returned by the aws emr list-instances --cluster-id command).

  • Type the following command to terminate an instance in the core instance group.

    aws emr modify-instance-groups --instance-groups InstanceGroupId=ig-6RXXXXXX07SA,EC2InstanceIdsToTerminate=i-f9XXXXf2

    For more information on using Amazon EMR commands in the AWS CLI, see http://docs.aws.amazon.com/cli/latest/reference/emr.

To add a task instance group with Spot Instances to a cluster using the Amazon EMR CLI

Note

The Amazon EMR CLI is no longer under feature development. Customers are encouraged to use the Amazon EMR commands in the AWS CLI instead.

  • You can use --add-instance-group to add a task instance group of Spot Instances to a running cluster. In the directory where you installed the Amazon EMR CLI, type the following command. For more information, see the Command Line Interface Reference for Amazon EMR.

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --jobflow JobFlowId \
      --add-instance-group task  --instance-type m1.small \
      --instance-count 5 --bid-price 0.05
    • Windows users:

      ruby elastic-mapreduce --jobflow JobFlowId --add-instance-group task  --instance-type m1.small --instance-count 5 --bid-price 0.05

To change the number of Spot Instances in instance groups using the Amazon EMR CLI

Note

The Amazon EMR CLI is no longer under feature development. Customers are encouraged to use the Amazon EMR commands in the AWS CLI instead.

  • You can change the number of requested Spot Instances in a cluster using the --modify-instance-group and --instance-count commands. Note that you can only increase the number of core instances in your cluster while you can increase or decrease the number of task instances. Setting the number of task instances to zero removes all Spot Instances (but not the instance group). In the directory where you installed the Amazon EMR CLI, type the following command to increase the number of requested Spot Instances for the task instance group from 4 to 5. For more information, see the Command Line Interface Reference for Amazon EMR.

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --jobflow JobFlowId \
      --modify-instance-group task --instance-count 5
    • Windows users:

      ruby elastic-mapreduce --jobflow JobFlowId --modify-instance-group task --instance-count 5

Add Spot Instances to Instance Groups Using the Java SDK

To add Spot Instances to a cluster using the Java SDK

  • To specify that an instance group should be launched as Spot Instances, set the withBidPrice and withMarket properties on the InstanceGroupConfig object that you instantiate for the instance group. The following code creates a task instance group of type m1.large with an instance count of 10. It specifies $0.35 as the maximum bid price, and runs as Spot Instances. When you make the call to modify the instance group, pass this object instance:

    InstanceGroupConfig instanceGroupConfig = new InstanceGroupConfig()
    	.withInstanceCount(10)
    	.withInstanceRole(“TASK”)
    	.withInstanceType(“m1.large”)
    	.withMarket("SPOT")
    	.withBidPrice(“0.35”);
    					

Add Spot Instances to Instance Groups Using the API

To add Spot Instances to a running cluster using the API

  • The following sample request increases the number of task nodes in the task instance group to eight and requests that they be launched as Spot Instances with an hourly bid price of .35. The following is an example of the request you would send to Amazon EMR.

Sample Request

https://elasticmapreduce.amazonaws.com?Operation=AddInstanceGroups
&InstanceGroups.member.1.InstanceGroupId=i-3UN6WX5RRO2AG 
&InstanceGroups.member.1.InstanceRequestCount=8
&InstanceGroups.member.1.InstanceRole=TASK
&InstanceGroups.member.1.Market=SPOT
&InstanceGroups.member.1.BidPrice=.35
&AuthParams

Sample Response

<ModifyInstanceGroupsResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31">
   <ResponseMetadata>
      <RequestId>
         2690d7eb-ed86-11dd-9877-6fad448a8419
      </RequestId>
   </ResponseMetadata>
</ModifyInstanceGroupsResponse>