View cluster status and details
After you create a cluster, you can monitor its status and get detailed information about
its execution and errors that may have occurred, even after it has terminated. Amazon EMR saves
metadata about terminated clusters for your reference for two months, after which the
metadata is deleted. You can't delete clusters from the cluster history, but using the
AWS Management Console, you can use the Filter, and using the AWS CLI, you can use
options with the list-clusters
command to focus on the clusters that you care
about.
You can access application history stored on-cluster for one week from the time it is recorded, regardless of whether the cluster is running or terminated. In addition, persistent application user interfaces store application history off-cluster for 30 days after a cluster terminates. See View application history.
For more information about cluster states, such as Waiting and Running, see Understanding the cluster lifecycle.
View cluster details using the AWS Management Console
The Clusters list in the https://console.aws.amazon.com/emr/
We’ve redesigned the Amazon EMR console to make it easier to use. See What's new with the console? to learn about the differences between the old and new console experiences.
Tab | Information |
---|---|
Properties |
Use this tab to view your cluster's operating system, your cluster termination and security configurations, your VPC and subnet information, and where you store logs in Amazon S3. |
Bootstrap actions |
Use this tab to view the status of any bootstrap actions the cluster runs when it launches. Bootstrap actions are used for custom software installations and advanced configuration. For more information, see Create bootstrap actions to install additional software. |
Monitoring |
Use this tab to view key metrics of cluster operation. You can view cluster-level data, node-level data, and information about I/O and data storage. |
Instances |
Use this tab to view information about nodes in your cluster, including EC2 instance IDs, DNS names, EBS volumes, and more. |
Steps |
Use this tab to see the status and access log files for steps that you submitted. For more information about steps, see Submit work to a cluster. |
Applications |
Use this tab to view persistent off-cluster YARN timeline server and Tez UI application details. You can also view information about your installed applications, cluster configurations, and instance groups. On-cluster application user interfaces are available while the cluster is running. |
Events |
Use this tab to view the event log for your cluster. For more information, see Monitor CloudWatch events. |
Tags |
Use this tab to view any tags you applied to the cluster. |
View cluster details using the AWS CLI
The following examples demonstrate how to retrieve cluster details using the AWS CLI.
For more information about available commands, see the AWS CLI Command Reference for Amazon EMR. You can use the describe-cluster command to
view cluster-level details including status, hardware and software configuration, VPC
settings, bootstrap actions, instance groups, and so on. For more information about
cluster states, see Understanding the cluster
lifecycle. The following example demonstrates
using the describe-cluster
command, followed by examples of the list-clusters command.
Example Viewing cluster status
To use the describe-cluster
command, you need the cluster ID. This example demonstrates using to get a list of clusters created within a certain date range, and then using one of the cluster IDs returned to list more information about an individual cluster's status.
The following command describes cluster j-1K48XXXXXXHCB
, which you replace with your cluster ID.
aws emr describe-cluster --cluster-id
j-1K48XXXXXXHCB
The output of your command is similar to the following:
{ "Cluster": { "Status": { "Timeline": { "ReadyDateTime": 1438281058.061, "CreationDateTime": 1438280702.498 }, "State": "WAITING", "StateChangeReason": { "Message": "Waiting for steps to run" } }, "Ec2InstanceAttributes": { "EmrManagedMasterSecurityGroup": "sg-cXXXXX0", "IamInstanceProfile": "EMR_EC2_DefaultRole", "Ec2KeyName": "myKey", "Ec2AvailabilityZone": "us-east-1c", "EmrManagedSlaveSecurityGroup": "sg-example" }, "Name": "Development Cluster", "ServiceRole": "EMR_DefaultRole", "Tags": [], "TerminationProtected": false, "ReleaseLabel": "emr-4.0.0", "NormalizedInstanceHours": 16, "InstanceGroups": [ { "RequestedInstanceCount": 1, "Status": { "Timeline": { "ReadyDateTime": 1438281058.101, "CreationDateTime": 1438280702.499 }, "State": "RUNNING", "StateChangeReason": { "Message": "" } }, "Name": "CORE", "InstanceGroupType": "CORE", "Id": "ig-2EEXAMPLEXXP", "Configurations": [], "InstanceType": "m5.xlarge", "Market": "ON_DEMAND", "RunningInstanceCount": 1 }, { "RequestedInstanceCount": 1, "Status": { "Timeline": { "ReadyDateTime": 1438281023.879, "CreationDateTime": 1438280702.499 }, "State": "RUNNING", "StateChangeReason": { "Message": "" } }, "Name": "MASTER", "InstanceGroupType": "MASTER", "Id": "ig-2A1234567XP", "Configurations": [], "InstanceType": "m5.xlarge", "Market": "ON_DEMAND", "RunningInstanceCount": 1 } ], "Applications": [ { "Version": "1.0.0", "Name": "Hive" }, { "Version": "2.6.0", "Name": "Hadoop" }, { "Version": "0.14.0", "Name": "Pig" }, { "Version": "1.4.1", "Name": "Spark" } ], "BootstrapActions": [], "MasterPublicDnsName": "ec2-X-X-X-X.compute-1.amazonaws.com", "AutoTerminate": false, "Id": "j-jobFlowID", "Configurations": [ { "Properties": { "hadoop.security.groups.cache.secs": "250" }, "Classification": "core-site" }, { "Properties": { "mapreduce.tasktracker.reduce.tasks.maximum": "5", "mapred.tasktracker.map.tasks.maximum": "2", "mapreduce.map.sort.spill.percent": "90" }, "Classification": "mapred-site" }, { "Properties": { "hive.join.emit.interval": "1000", "hive.merge.mapfiles": "true" }, "Classification": "hive-site" } ] } }
Example Listing clusters by creation date
To retrieve clusters created within a specific data range, use the list-clusters
command with the
--created-after
and --created-before
parameters.
The following command lists all clusters created between October 09, 2019 and October 12, 2019.
aws emr list-clusters --created-after
2019-10-09T00:12:00
--created-before2019-10-12T00:12:00
Example Listing clusters by state
To list clusters by state, use the list-clusters
command with the
--cluster-states
parameter. Valid cluster states include: STARTING,
BOOTSTRAPPING, RUNNING, WAITING, TERMINATING, TERMINATED, and
TERMINATED_WITH_ERRORS.
aws emr list-clusters --cluster-states
TERMINATED
You can also use the following shortcut parameters to list all clusters in the states specified.:
--active
filters clusters in the STARTING,BOOTSTRAPPING, RUNNING, WAITING, or TERMINATING states.--terminated
filters clusters in the TERMINATED state.--failed
parameter filters clusters in the TERMINATED_WITH_ERRORS state.
The following commands return the same result.
aws emr list-clusters --cluster-states TERMINATED
aws emr list-clusters --terminated
For more information about cluster states, see Understanding the cluster lifecycle.