Amazon Elastic MapReduce
Developer Guide (API Version 2009-03-31)
« PreviousNext »
View the PDF for this guide.Go to the AWS Discussion Forum for this product.Go to the Kindle Store to download this guide in Kindle format.Did this page help you?  Yes | No |  Tell us about it...

Resize a Running Cluster

A cluster contains a single master node. The master node controls any slave nodes that are present. There are two types of slave nodes: core nodes, which store data in the Hadoop Distributed File System (HDFS), and task nodes, which do not use HDFS.

Nodes within a cluster are managed as instance groups. All clusters require a master instance group containing a single master node. Clusters using slave nodes require a core instance group that contains at least one core node. Additionally, if a cluster has a core instance group, it can also have a task instance group containing one or more task nodes.

Note

You must have at least one core node at cluster creation in order to resize the cluster. In other words, single node clusters cannot be resized.

You can resize the core instance group in a running cluster by adding nodes using the console, CLI, or API. You cannot shrink the size of the core instance group in a running cluster by reducing the instance count. However, it is possible to terminate an instance in the core instance group using the AWS CLI or the API. This should be done with caution. Terminating an instance in the core instance group risks data loss, and the instance is not automatically replaced.

Task nodes also run your Hadoop jobs. After a cluster is running, you can increase or decrease the number of task nodes uisng the console, CLI, or API.

When your cluster runs, Hadoop determines the number of mapper and reducer tasks needed to process the data. Larger clusters should have more tasks for better resource use and shorter processing time. Typically, an Amazon EMR cluster remains the same size during the entire cluster; you set the number of tasks when you create the cluster. When you resize a running cluster, you can vary the processing during the cluster execution. Therefore, instead of using a fixed number of tasks, you can vary the number of tasks during the life of the cluster. There are two configuration options to help set the ideal number of tasks.

  • mapred.map.tasksperslot

  • mapred.reduce.tasksperslot

You can set both options in the mapred-conf.xml file. When you submit a job to the cluster, the job client checks the current total number of map and reduce slots available cluster wide. The job client then uses the following equations to set the number of tasks:

  • mapred.map.tasks = mapred.map.tasksperslot * map slots in cluster

  • mapred.reduce.tasks = mapred.reduce.tasksperslot * reduce slots in cluster

The job client only reads the tasksperslot parameter if the number of tasks is not configured. You can override the number of tasks at any time, either for all clusters via a bootstrap action or individually per job by adding a step to change the configuration.

Amazon EMR withstands slave node failures and continues cluster execution even if a slave node becomes unavailable. Amazon EMR automatically provisions additional slave nodes to replace those that fail.

You can have a different number of slave nodes for each cluster step. You can also add a step to a running cluster to modify the number of slave nodes. Because all steps are guaranteed to run sequentially by default, you can specify the number of running slave nodes for any step.

Resize a Cluster Using the Console

You can use the Amazon EMR console to resize a running cluster.

To resize a running cluster using the console

  1. From the Cluster List page, click a cluster to resize.

  2. On the Cluster Details page, click Resize. Alternatively, you can expand the Hardware Configuration section, click the Resize button adjacent to the core or task nodes, and increase or decrease the number of instances for the instance group.

  3. To add task nodes to a cluster that has none, click Add Task Nodes to choose the task node type, the number of task nodes, and whether the task nodes are spot instances.

Note

You can only increase the number of core nodes in the console, but you can both increase and decrease the number of task nodes.

When you make a change to the number of nodes, the Amazon EMR console updates the status of the instance group through the Provisioning and Resizing states until they are ready and indicate in brackets the newly requested number of nodes. When the change to the node count finishes, the instance groups return to the Running state.

Resize a Cluster Using the AWS CLI or the Amazon EMR CLI

You can use either the AWS CLI or the Amazon EMR CLI to resize a running cluster. You can increase or decrease the number of task nodes, and you can increase the number of core nodes in a running cluster. It is also possible to terminate an instance in the core instance group using the AWS CLI or the API. This should be done with caution. Terminating an instance in the core instance group risks data loss, and the instance is not automatically replaced.

In addition to resizing the core and task groups, you can also add a task group to a running cluster using the AWS CLI or the Amazon EMR CLI.

To resize a cluster by changing the instance count using the AWS CLI

  • You can add instances to the core group or task group, and you can remove instances from the task group using the AWS CLI modify-instance-groups subcommand with the InstanceCount parameter:

    aws emr modify-instance-groups --instance-groups InstanceGroupId=string,InstanceCount=integer

    To add instances to the core or task groups, increase the InstanceCount. To reduce the number of instances in the task group, decrease the InstanceCount. Changing the instance count of the task group to 0 removes all instances but not the instance group. For example, to increase the number of instances in the task instance group from 3 to 4, type:

    aws emr modify-instance-groups --instance-groups InstanceGroupId=ig-31JXXXXXXBTO,InstanceCount=4

    To retrieve the InstanceGroupId, use the describe-cluster subcommand. The output is a JSON object called Cluster that contains the ID of each instance group. To use this command, you need the cluster ID (which you can retrieve using the aws emr list-clusters command or the console):

    aws emr describe-cluster --cluster-id string

    Using the AWS CLI, you can also terminate an instance in the core instance group with the --modify-instance-groups subcommand. This should be done with caution. Terminating an instance in the core instance group risks data loss, and the instance is not automatically replaced. To terminate a specific instance you need the instance group ID (returned by the aws emr describe-cluster --cluster-id subcommand) and the instance ID (returned by the aws emr list-instances --cluster-id subcommand):

    aws emr modify-instance-groups --instance-groups InstanceGroupId=string,EC2InstanceIdsToTerminate=string

    For example:

    aws emr modify-instance-groups --instance-groups InstanceGroupId=ig-6RXXXXXX07SA,EC2InstanceIdsToTerminate=i-f9XXXXf2

    For more information on using Amazon EMR commands in the AWS CLI, see http://docs.aws.amazon.com/cli/latest/reference/emr.

To resize a cluster by adding a task instance group using the AWS CLI

  • Using the AWS CLI, you can add a task instance group to a cluster with the --add-instance-groups subcommand:

    aws emr add-instance-groups --cluster-id string --instance-groups InstanceCount=integer,InstanceGroupType=string,InstanceType=string

    For example:

    aws emr add-instance-groups --cluster-id j-JXBXXXXXX37R --instance-groups InstanceCount=6,InstanceGroupType=task,InstanceType=m1.large

    Note

    You cannot add a task instance group to a cluster if the cluster was created with a task group, even if the instance count for the group is 0.

    For more information on using Amazon EMR commands in the AWS CLI, see http://docs.aws.amazon.com/cli/latest/reference/emr.

Parameters to increase or decrease nodes using the Amazon EMR CLI

Note

The Amazon EMR CLI is no longer under feature development. Customers are encouraged to use the Amazon EMR commands in the AWS CLI instead.

  • You can increase or decrease the number of task nodes in a running cluster using the Amazon EMR CLI. You can also increase but not decrease the number of core nodes. The parameters are listed in the following table.

    ParameterDescription
    --modify-instance-group INSTANCE_GROUP_IDModify an existing instance group.
    --instance-count INSTANCE_COUNT

    Set the count of nodes for an instance group.

    Note

    You are only allowed to increase the number of nodes in a core instance group.

    You can increase or decrease the number of nodes in a task instance group.

    Master instance groups can not be modified.

Parameters to add a task instance group to a running cluster using the Amazon EMR CLI

Note

The Amazon EMR CLI is no longer under feature development. Customers are encouraged to use the Amazon EMR commands in the AWS CLI instead.

  • You can add a task instance group to your running cluster using the Amazon EMR CLI. The parameters are listed in the following table.

    ParameterDescription
    --add-instance-group ROLEAdd an instance group to an existing cluster. The role may be TASK only. Currently, Amazon Elastic MapReduce (Amazon EMR) does not permit adding core or master instance groups to a running cluster.
    --instance-count INSTANCE_COUNTSet the count of nodes for an instance group.
    --instance-type INSTANCE_TYPESet the type of EC2 instance to create nodes for an instance group.

Arrested State

An instance group goes into arrested state if it encounters too many errors while trying to start the new cluster nodes. For example, if new nodes fail while performing bootstrap actions, the instance group goes into an ARRESTED state, rather than continuously provisioning new nodes. After you resolve the underlying issue, reset the desired number of nodes on the cluster's instance group, and then the instance group resumes allocating nodes. Modifying an instance group instructs Amazon EMR to attempt to provision nodes again. No running nodes are restarted or terminated.

In the AWS CLI, the list-instances subcommand returns all instances and their states as does the describe-cluster subcommand. In the Amazon EMR CLI, the --describe command returns all instance groups and node types, and you can see the state of the instance groups for the cluster. If Amazon EMR detects a fault with an instance group, it changes the group's state to ARRESTED.

To reset a cluster in an ARRESTED state using the AWS CLI

  • Type the describe-cluster subcommand with the --cluster-id parameter to view the state of the instances in your cluster.

    For example, to view information on all instances and instance groups in a cluster, type:

    aws emr describe-cluster --cluster-id j-3KVXXXXXXY7UG

    The output will display information about your instance groups and the state of the instances:

    {
        "Cluster": {
            "Status": {
                "Timeline": {
                    "ReadyDateTime": 1413187781.245,
                    "CreationDateTime": 1413187405.356
                },
                "State": "WAITING",
                "StateChangeReason": {
                    "Message": "Waiting after step completed"
                }
            },
            "Ec2InstanceAttributes": {
                "Ec2AvailabilityZone": "us-west-2b"
            },
            "Name": "Development Cluster",
            "Tags": [],
            "TerminationProtected": false,
            "RunningAmiVersion": "3.1.1",
            "NormalizedInstanceHours": 16,
            "InstanceGroups": [
                {
                    "RequestedInstanceCount": 1,
                    "Status": {
                        "Timeline": {
                            "ReadyDateTime": 1413187775.749,
                            "CreationDateTime": 1413187405.357
                        },
                        "State": "RUNNING",
                        "StateChangeReason": {
                            "Message": ""
                        }
                    },
                    "Name": "MASTER",
                    "InstanceGroupType": "MASTER",
                    "InstanceType": "m1.large",
                    "Id": "ig-3ETXXXXXXFYV8",
                    "Market": "ON_DEMAND",
                    "RunningInstanceCount": 1
                },
                {
                    "RequestedInstanceCount": 1,
                    "Status": {
                        "Timeline": {
                            "ReadyDateTime": 1413187781.301,
                            "CreationDateTime": 1413187405.357
                        },
                        "State": "RUNNING",
                        "StateChangeReason": {
                            "Message": ""
                        }
                    },
                    "Name": "CORE",
                    "InstanceGroupType": "CORE",
                    "InstanceType": "m1.large",
                    "Id": "ig-3SUXXXXXXQ9ZM",
                    "Market": "ON_DEMAND",
                    "RunningInstanceCount": 1
                }
    ...
    }

    To view information on a particular instance group, type the list-instances subcommand with the --cluster-id and --instance-group-types parameters. You can view information for the MASTER, CORE, or TASK groups:

    aws emr list-instances --cluster-id j-3KVXXXXXXY7UG --instance-group-types "CORE"

    Use the modify-instance-groups subcommand with the --instance-groups parameter to reset a cluster in the ARRESTED state. The instance group id is returned by the describe-cluster subcommand:

    aws emr modify-instance-groups --instance-groups InstanceGroupId=string,InstanceCount=integer

    For example:

    aws emr modify-instance-groups --instance-groups InstanceGroupId=ig-3SUXXXXXXQ9ZM,InstanceCount=3

Tip

You do not need to change the number of nodes from the original configuration to free a running cluster. Set -–instance-count to the same count as the original setting.

To reset a cluster in an ARRESTED state using the Amazon EMR CLI

Note

The Amazon EMR CLI is no longer under feature development. Customers are encouraged to use the Amazon EMR commands in the AWS CLI instead.

  • Use the --modify-instance-group command to reset a cluster in the ARRESTED state. Enter the --modify-instance-group command as follows:

    In the directory where you installed the Amazon EMR CLI, run the following from the command line. For more information, see the Command Line Interface Reference for Amazon EMR.

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --modify-instance-group InstanceGroupID \
      -–instance-count COUNT
    • Windows users:

      ruby elastic-mapreduce --modify-instance-group InstanceGroupID -–instance-count COUNT

    The <InstanceGroupID> is the ID of the arrested instance group and <COUNT> is the number of nodes you want in the instance group.

Tip

You do not need to change the number of nodes from the original configuration to free a running cluster. Set -–instance-count to the same count as the original setting.

Legacy Clusters

Before October 2010, Amazon EMR did not have the concept of instance groups. Clusters developed for Amazon EMR that were built before the option to resize running clusters was available are considered legacy clusters. Previously, the Amazon EMR architecture did not use instance groups to manage nodes and only one type of slave node existed. Legacy clusters reference slaveInstanceType and other now deprecated fields. Amazon EMR continues to support the legacy clusters; you do not need to modify them to run them correctly.

Cluster Behavior

If you run a legacy cluster and only configure master and slave nodes, you observe a slaveInstanceType and other deprecated fields associated with your clusters.

Mapping Legacy Clusters to Instance Groups

Before October 2010, all cluster nodes were either master nodes or slave nodes. An Amazon EMR configuration could typically be represented like the following diagram.

Old Amazon EMR Model

1 A legacy cluster launches and a request is sent to Amazon EMR to start the cluster.
2 Amazon EMR creates a Hadoop cluster.
3 The legacy cluster runs on a cluster consisting of a single master node and the specified number of slave nodes.

Clusters created using the older model are fully supported and function as originally designed. The Amazon EMR API and commands map directly to the new model. Master nodes remain master nodes and become part of the master instance group. Slave nodes still run HDFS and become core nodes and join the core instance group.

Note

No task instance group or task nodes are created as part of a legacy cluster, however you can add them to a running cluster at any time.

The following diagram illustrates how a legacy cluster now maps to master and core instance groups.

Old Amazon EMR Model Remapped to Current Architecture

1 A request is sent to Amazon EMR to start a cluster.
2 Amazon EMR creates an Hadoop cluster with a master instance group and core instance group.
3 The master node is added to the master instance group.
4 The slave nodes are added to the core instance group.