Using termination protection
Termination protection protects your clusters from accidental termination, which can be especially useful for long running clusters processing critical workloads. When termination protection is enabled on a long-running cluster, you can still terminate the cluster, but you must explicitly remove termination protection from the cluster first. This helps ensure that EC2 instances are not shut down by an accident or error. You can enable termination protection when you create a cluster, and you can change the setting on a running cluster.
With termination protection enabled, the TerminateJobFlows
action in the
Amazon EMR API does not work. Users cannot terminate the cluster using this API or the
terminate-clusters
command from the AWS CLI. The API returns an error,
and the CLI exits with a non-zero return code. When you use the Amazon EMR console to
terminate a cluster, you are prompted with an extra step to turn termination protection
off.
Warning
Termination protection does not guarantee that data is retained in the event of a human error or a workaround—for example, if a reboot command is issued from the command line while connected to the instance using SSH, if an application or script running on the instance issues a reboot command, or if the Amazon EC2 or Amazon EMR API is used to disable termination protection. This is true as well if you're running Amazon EMR releases 7.1 and higher and an instance becomes unhealthy and unrecoverable. Even with termination protection enabled, data saved to instance storage, including HDFS data, can be lost. Write data output to Amazon S3 locations and create backup strategies as appropriate for your business continuity requirements.
Termination protection does not affect your ability to scale cluster resources using any of the following actions:
-
Resizing a cluster manually with the AWS Management Console or AWS CLI. For more information, see Manually resizing a running cluster.
-
Removing instances from a core or task instance group using a scale-in policy with automatic scaling. For more information, see Using automatic scaling with a custom policy for instance groups.
-
Removing instances from an instance fleet by reducing target capacity. For more information, see Instance fleet options.
Termination protection and Amazon EC2
An Amazon EMR cluster with termination protection enabled has the
disableAPITermination
attribute set for all Amazon EC2 instances in the
cluster. If a termination request originates with Amazon EMR, and the Amazon EMR and Amazon EC2
settings for an instance conflict, the Amazon EMR setting overrides the Amazon EC2 setting.
For example, if you use the Amazon EC2 console to enable termination
protection on an Amazon EC2 instance in a cluster that has termination protection
disabled, when you use the Amazon EMR console, AWS CLI commands
for Amazon EMR, or the Amazon EMR API to terminate the cluster, Amazon EMR sets
DisableApiTermination
to false
and terminates the
instance along with other instances.
Important
If an instance is created as part of an Amazon EMR cluster with termination
protection, and the Amazon EC2 API or AWS CLI commands are used to modify the instance
so that DisableApiTermination
is false
, and then the
Amazon EC2 API or AWS CLI commands execute the TerminateInstances
action,
the Amazon EC2 instance terminates.
Termination protection and unhealthy YARN nodes
Amazon EMR periodically checks the Apache Hadoop YARN status of nodes running on core
and task Amazon EC2 instances in a cluster. The health status is reported by the NodeManager health checker serviceUNHEALTHY
, the Amazon EMR instance controller adds the node to a denylist and
does not allocate YARN containers to it until it becomes healthy again. Depending on the statuses of termination protection,
unhealthy node replacement, and Amazon EMR release version, Amazon EMR will either
replace the unhealthy instance or stop allocating controllers to the instance.
Termination protection and termination after step execution
When you enable termination after step execution and also enable termination protection, Amazon EMR ignores the termination protection.
When you submit steps to a cluster, you can set the ActionOnFailure
property to determine what happens if the step can't complete execution because of
an error. The possible values for this setting are TERMINATE_CLUSTER
(TERMINATE_JOB_FLOW
with earlier versions),
CANCEL_AND_WAIT
, and CONTINUE
. For more information,
see Submit work to a cluster.
If a step fails that is configured with ActionOnFailure
set to
CANCEL_AND_WAIT
, if termination after step execution is enabled, the cluster
terminates without executing subsequent steps.
If a step fails that is configured with ActionOnFailure
set to
TERMINATE_CLUSTER
, use the table of settings below to determine the
outcome.
ActionOnFailure | Termination after step execution | Termination protection | Result |
---|---|---|---|
|
Enabled |
Disabled |
Cluster terminates |
Enabled |
Enabled |
Cluster terminates |
|
Disabled |
Enabled |
Cluster continues |
|
Disabled |
Disabled |
Cluster terminates |
Termination protection and Spot Instances
Amazon EMR termination protection does not prevent an Amazon EC2 Spot Instance from terminating when the Spot price rises above the maximum Spot price.
Configuring termination protection when you launch a cluster
You can enable or disable termination protection when you launch a cluster using the console, the AWS CLI, or the API.
For single-node clusters, default termination protection settings are as follows:
-
Launching a cluster by Amazon EMR Console —Termination Protection is disabled by default.
-
Launching a cluster by AWS CLI
aws emr create-cluster
—Termination Protection is disabled unless--termination-protected
is specified. -
Launching a cluster by Amazon EMR API RunJobFlow command—Termination Protection is disabled unless the
TerminationProtected
boolean value is set totrue
.
For high-availability clusters, default termination protection settings are as follows:
-
Launching a cluster by Amazon EMR Console — Termination Protection is enabled by default.
-
Launching a cluster by AWS CLI
aws emr create-cluster
—Termination Protection is disabled unless--termination-protected
is specified. -
Launching a cluster by Amazon EMR API RunJobFlow command—Termination Protection is disabled unless the
TerminationProtected
boolean value is set totrue
.
Configuring termination protection for running clusters
You can configure termination protection for a running cluster with the console or the AWS CLI.