Amazon Elastic MapReduce
Developer Guide (API Version 2009-03-31)
Did this page help you?  Yes | No |  Tell us about it...
« PreviousNext »
View the PDF for this guide.Go to the AWS Discussion Forum for this product.Go to the Kindle Store to download this guide in Kindle format.

Managing Cluster Termination

Termination protection ensures that the EC2 instances in your job flow are not shut down by an accident or error. This protection is especially useful if your cluster contains data in instance storage that you need to recover before those instances are terminated.

When termination protection is not enabled, you can terminate clusters either through calls to the TerminateJobFlows API, through the Amazon EMR console, or by using the command line interface. In addition, the master node may terminate a task node that has become unresponsive or has returned an error.

By default, termination protection is enabled when you launch a cluster using the console. Termination proctection is disabled by default when you launch a cluster using the CLI or API. When termination protection is enabled, you must explicitly remove termination protection from the cluster before you can terminate it. With termination protection enabled, TerminateJobFlows cannot terminate the cluster and users cannot terminate the cluster using the CLI. Users terminating the cluster using the Amazon EMR console receive an extra confirmation box asking if they want to remove termination protection before terminating the cluster.

If you attempt to terminate a protected cluster with the API or CLI, the API returns an error, and the CLI exits with a non-zero return code.

When you submit steps to a cluster, the ActionOnFailure setting determines what the cluster does in response to any errors. The possible values for this setting are:

  • TERMINATE_JOB_FLOW: If the step fails, terminate the cluster. If the cluster has termination protection enabled AND auto-terminate disabled, it will not terminate.

  • CANCEL_AND_WAIT: If the step fails, cancel the remaining steps. If the cluster has auto-terminate disabled, the cluster will not terminate.

  • CONTINUE: If the step fails, continue to the next step.

Termination Protection in Amazon EMR and Amazon EC2

Termination protection of clusters in Amazon EMR is analogous to setting the disableAPITermination flag on an EC2 instance. In the event of a conflict between the termination protection set in Amazon EC2 and that set in Amazon EMR, the Amazon EMR cluster protection status overrides that set by Amazon EC2 on the given instance. For example, if you use the Amazon EC2 console to enable termination protection on an EC2 instance in an Amazon EMR cluster that has termination protection disabled, Amazon EMR turns off termination protection on that EC2 instance and shuts down the instance when the rest of the cluster terminates.

Termination Protection and Spot Instances

Amazon EMR termination protection does not prevent an Amazon EC2 Spot Instance from terminating when the Spot Price rises above the maximum bid price. For more information about the behavior of Spot Instances in Amazon EMR, see (Optional) Lower Costs with Spot Instances.

Termination Protection and Auto-terminate

Enabling auto-terminate creates a transient cluster. The cluster automatically terminates when the last step is successfully completed even if termination protection is enabled.

Disabling auto-terminate causes instances in a cluster to persist after steps have successfully completed, but still allows the cluster to be terminated by user action, by errors, and by calls to TerminateJobFlows (if termination protection is disabled).

Note

By default, auto-terminate is disabled for clusters launched using the console and the CLI. Clusters launched using the API have auto-terminate enabled.

Configuring Termination Protection for New Clusters

You can enable or disable termination protection when you launch a cluster using the console, the AWS CLI, the Amazon EMR CLI, or the API.

To configure termination protection for a new cluster using the console

  1. Open the Amazon Elastic MapReduce console at https://console.aws.amazon.com/elasticmapreduce/.

  2. Click Create cluster.

  3. In the Cluster Configuration section, set the Termination protection field to Yes to enable protection, or set the field to No to disable it. By default, termination protection is enabled.

    Termination protection setting
  4. Proceed with creating the cluster as described in Plan an Amazon EMR Cluster.

To configure termination protection for a new cluster using the AWS CLI

Using the AWS CLI, you can launch a cluster with termination proctection enabled by typing the create-cluster command with the --termination-protected parameter. By default, termination protection is disabled when you launch a cluster using the AWS CLI. You can also use the --no-termination-protected parameter to disable termination protection.

  • To launch a protected cluster, type the following command and replace myKey with the name of your EC2 key pair.

    • Linux, UNIX, and Mac OS X users:

      aws emr create-cluster --name "Test cluster" --ami-version 3.3 --applications Name=Hue Name=Hive Name=Pig \
      --use-default-roles --ec2-attributes KeyName=myKey \
      --instance-type m3.xlarge --instance-count 3 --termination-protected
    • Windows users:

      aws emr create-cluster --name "Test cluster" --ami-version 3.3 --applications Name=Hue Name=Hive Name=Pig --use-default-roles --ec2-attributes KeyName=myKey --instance-type m3.xlarge --instance-count 3 --termination-protected

    For more information on using Amazon EMR commands in the AWS CLI, see http://docs.aws.amazon.com/cli/latest/reference/emr.

To configure termination protection for a new cluster using the Amazon EMR CLI

Note

The Amazon EMR CLI is no longer under feature development. Customers are encouraged to use the Amazon EMR commands in the AWS CLI instead.

  • To enable termination protection using the Amazon EMR CLI, specify --set-termination-protection true during the cluster creation call. If the parameter is not used, termination protection is disabled. You can also type --set-termination-protection false to disable protection. The following example shows setting termination protection on a cluster running the WordCount sample application.

    In the directory where you installed the Amazon EMR CLI, type the following command. For more information, see the Command Line Interface Reference for Amazon EMR.

    Note

    The Hadoop streaming syntax shown in the following examples is different between Hadoop 1.x and Hadoop 2.x.

    For Hadoop 2.x, type the following command:

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --create --alive --ami-version 3.0.3 \
      --instance-type m1.xlarge --num-instances 2 \
      --stream --arg "-files" --arg "s3://elasticmapreduce/samples/wordcount/wordSplitter.py" \
      --input s3://elasticmapreduce/samples/wordcount/input \
      --output s3://mybucket/output/2014-01-16 --mapper wordSplitter.py --reducer aggregate \
      --set-termination-protection true
    • Windows users:

      ruby elastic-mapreduce --create --alive --ami-version 3.0.3 --instance-type m1.xlarge --num-instances 2 --stream --arg "-files" --arg "s3://elasticmapreduce/samples/wordcount/wordSplitter.py" --input s3://elasticmapreduce/samples/wordcount/input --output s3://mybucket/output/2014-01-16 --mapper wordSplitter.py --reducer aggregate --set-termination-protection true

    For Hadoop 1.x, type the following command:

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --create --alive /
      --instance-type m1.xlarge --num-instances 2 --stream /
      --input s3://elasticmapreduce/samples/wordcount/input /
      --output s3://myawsbucket/wordcount/output/2011-03-25 /
      --mapper s3://elasticmapreduce/samples/wordcount/wordSplitter.py --reducer aggregate /
      --set-termination-protection true
    • Windows users:

      ruby elastic-mapreduce --create --alive --instance-type m1.xlarge --num-instances 2 --stream --input s3://elasticmapreduce/samples/wordcount/input --output s3://myawsbucket/wordcount/output/2011-03-25 --mapper s3://elasticmapreduce/samples/wordcount/wordSplitter.py --reducer aggregate --set-termination-protection true

Configuring Termination Protection for Running Clusters

You can configure termination protection for a running cluster using the console, the Amazon EMR CLI, or the AWS CLI.

To configure termination protection for a running cluster using the console

  1. Open the Amazon Elastic MapReduce console at https://console.aws.amazon.com/elasticmapreduce/.

  2. On the Cluster List page, click the link for your cluster.

  3. On the Cluster Details page, in the Summary section, for Termination protection, click Change.

  4. Click On and then click the check mark icon to enable termination protection. Alternatively, click Off to disable it.

    Confirm termination protection change

To configure termination protection for a running cluster using the AWS CLI

To enable termination protection on a running cluster using the AWS CLI, type the modify-cluster-attributes subcommand with the --termination-protected parameter. To disable it, type the --no-termination-protected parameter.

  • Type the following command to enable termination protection on a running cluster.

    aws emr modify-cluster-attributes --cluster-id j-3KVTXXXXXX7UG --termination-protected

    To disable termination protection, type:

    aws emr modify-cluster-attributes --cluster-id j-3KVTXXXXXX7UG --no-termination-protected

To configure termination protection for a running cluster using the Amazon EMR CLI

Note

The Amazon EMR CLI is no longer under feature development. Customers are encouraged to use the Amazon EMR commands in the AWS CLI instead.

  • Set the --set-termination-protection flag to true. This is shown in the following example, where JobFlowID is the identifier of the cluster on which to enable termination protection.

    In the directory where you installed the Amazon EMR CLI, type the following command. For more information, see the Command Line Interface Reference for Amazon EMR.

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --set-termination-protection true --jobflow JobFlowID
    • Windows users:

      ruby elastic-mapreduce --set-termination-protection true --jobflow JobFlowID

    Disable termination protection by setting the --set-termination-protection parameter to false. This is shown in the following example, where JobFlowID is the identifier of the cluster on which to disable termination protection.

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --set-termination-protection false --jobflow JobFlowID
    • Windows users:

      ruby elastic-mapreduce --set-termination-protection false --jobflow JobFlowID