Amazon Elastic MapReduce
Developer Guide (API Version 2009-03-31)
Did this page help you?  Yes | No |  Tell us about it...
« PreviousNext »
View the PDF for this guide.Go to the AWS Discussion Forum for this product.Go to the Kindle Store to download this guide in Kindle format.

View Cluster Details

After you start a cluster, you can monitor its status and retrieve extended information about its execution. This section describes the methods used to view the details of Amazon EMR clusters. You can view clusters in any state.

This procedure explains how to view the details of a cluster using the Amazon EMR console.

To view cluster details using the console

  1. Sign in to the AWS Management Console and open the Amazon EMR console at https://console.aws.amazon.com/elasticmapreduce/.

  2. You have the option to view details in the Cluster List page. By selecting the arrow icon next to each cluster, the row view expands and gives some details and actions for the cluster you chose:

    Cluster information

  3. For more details, click the cluster link to optn the Cluster Details page. The Summary section displays detailed information about the selected cluster.

    Cluster details page

The following examples demonstrate how to retrieve cluster details using the AWS CLI and the Amazon EMR CLI.

To view cluster details using the AWS CLI

Use the describe-cluster command to view cluster-level details including status, hardware and software configuration, VPC settings, bootstrap actions, instance groups and so on.

  • To use the describe-cluster command, you need the cluster ID. The cluster ID can be retrieved using the list-clusters command. To view cluster details, type the following command and replace j-1K48XXXXXXHCB with your cluster ID.

    aws emr describe-cluster --cluster-id j-1K48XXXXXXHCB

    The output is similar to the following:

    {
        "Cluster": {
            "Status": {
                "Timeline": {
                    "ReadyDateTime": 1413102659.072,
                    "EndDateTime": 1413103872.89,
                    "CreationDateTime": 1413102441.707
                },
                "State": "TERMINATED",
                "StateChangeReason": {
                    "Message": "Terminated by user request",
                    "Code": "USER_REQUEST"
                }
            },
            "Ec2InstanceAttributes": {
                "Ec2AvailabilityZone": "us-west-2a"
            },
            "Name": "Development Cluster",
            "Tags": [],
            "TerminationProtected": false,
            "RunningAmiVersion": "3.1.0",
            "NormalizedInstanceHours": 24,
            "InstanceGroups": [
                {
                    "RequestedInstanceCount": 2,
                    "Status": {
                        "Timeline": {
                            "ReadyDateTime": 1413102659.09,
                            "EndDateTime": 1413103872.779,
                            "CreationDateTime": 1413102441.708
                        },
                        "State": "TERMINATED",
                        "StateChangeReason": {
                            "Message": "Job flow terminated",
                            "Code": "CLUSTER_TERMINATED"
                        }
                    },
                    "Name": "CORE",
                    "InstanceGroupType": "CORE",
                    "InstanceType": "m3.xlarge",
                    "Id": "ig-115XXXXXX52SX",
                    "Market": "ON_DEMAND",
                    "RunningInstanceCount": 0
                },
                {
                    "RequestedInstanceCount": 1,
                    "Status": {
                        "Timeline": {
                            "ReadyDateTime": 1413102655.968,
                            "EndDateTime": 1413103872.779,
                            "CreationDateTime": 1413102441.708
                        },
                        "State": "TERMINATED",
                        "StateChangeReason": {
                            "Message": "Job flow terminated",
                            "Code": "CLUSTER_TERMINATED"
                        }
                    },
                    "Name": "MASTER",
                    "InstanceGroupType": "MASTER",
                    "InstanceType": "m3.xlarge",
                    "Id": "ig-26LXXXXXXFCXQ",
                    "Market": "ON_DEMAND",
                    "RunningInstanceCount": 0
                }
            ],
            "Applications": [
                {
                    "Version": "2.4.0",
                    "Name": "hadoop"
                }
            ],
            "MasterPublicDnsName": "ec2-XX-XX-XXX-XX.us-west-2.compute.amazonaws.com
    ",
            "VisibleToAllUsers": true,
            "BootstrapActions": [
                {
                    "Args": [],
                    "Name": "Install Ganglia",
                    "ScriptPath": "s3://us-west-2.elasticmapreduce/bootstrap-actions
    /install-ganglia"
                }
            ],
            "RequestedAmiVersion": "3.1.0",
            "AutoTerminate": false,
            "Id": "j-Z2OXXXXXXI45"
        }
    }

    For more information on using Amazon EMR commands in the AWS CLI, see http://docs.aws.amazon.com/cli/latest/reference/emr.

To list clusters by creation date using the AWS CLI

To list clusters by creation date, type the list-clusters command with the --created-after and --created-before parameters.

  • To list clusters created between 10-09-2014 and 10-12-2014, type the following command.

    aws emr list-clusters --created-after 2014-10-09T00:12:00 --created-before 2014-10-12T00:12:00

    The output lists all clusters created between the specified timestamps.

    For more information on using Amazon EMR commands in the AWS CLI, see http://docs.aws.amazon.com/cli/latest/reference/emr.

To list clusters by state using the AWS CLI

To list clusters by state, type the list-clusters command with the --cluster-states parameter. Valid cluster states include: STARTING, BOOTSTRAPPING, RUNNING, WAITING, TERMINATING, TERMINATED, and TERMINATED_WITH_ERRORS.

You can also use several shortcut parameters to view clusters. The --active parameter filters clusters in the STARTING,BOOTSTRAPPING, RUNNING, WAITING, or TERMINATING states. The --terminated parameter filters clusters in the TERMINATED state. The --failed parameter filters clusters in the TERMINATED_WITH_ERRORS state.

  • Type the following command to list all TERMINATED clusters.

    aws emr list-clusters --cluster-states TERMINATED

    Or:

    aws emr list-clusters --terminated

    The output lists all clusters in the state specified.

    For more information on using Amazon EMR commands in the AWS CLI, see http://docs.aws.amazon.com/cli/latest/reference/emr.

To list clusters created in the last two days using the Amazon EMR CLI

Note

The Amazon EMR CLI is no longer under feature development. Customers are encouraged to use the Amazon EMR commands in the AWS CLI instead.

  • Use the --list parameter with no additional arguments to display clusters created during the last two days as follows:

    In the directory where you installed the Amazon EMR CLI, run the following from the command line. For more information, see the Command Line Interface Reference for Amazon EMR.

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --list
    • Windows users:

      ruby elastic-mapreduce --list

The response is similar to the following:

j-1YE2DN7RXJBWU   FAILED      Example Job Flow
                  CANCELLED   Custom Jar
j-3GJ4FRRNKGY97   COMPLETED   ec2-67-202-3-73.compute-1.amazonaws.com   Example cluster
j-5XXFIQS8PFNW    COMPLETED   ec2-67-202-51-30.compute-1.amazonaws.com  demo 3/24 s1
                  COMPLETED   Custom Jar 

The example response shows that three clusters were created in the last two days. The indented lines are the steps of the cluster. The information for a cluster is in the following order: the cluster ID, the cluster state, the DNS name of the master node, and the cluster name. The information for a cluster step is in the following order: step state, and step name.

If no clusters were created in the previous two days, this command produces no output.

To list active clusters using the Amazon EMR CLI

Note

The Amazon EMR CLI is no longer under feature development. Customers are encouraged to use the Amazon EMR commands in the AWS CLI instead.

  • Use the --list and --active parameters as follows:

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce  --list --active
    • Windows users:

      ruby elastic-mapreduce  --list --active

The response lists clusters that are in the state of STARTING, RUNNING, or SHUTTING_DOWN.

To list only running or terminated clusters using the Amazon EMR CLI

Note

The Amazon EMR CLI is no longer under feature development. Customers are encouraged to use the Amazon EMR commands in the AWS CLI instead.

  • Use the --state parameter as follows:

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --list --state RUNNING  --state TERMINATED
    • Windows users:

      ruby elastic-mapreduce --list --state RUNNING  --state TERMINATED

The response lists clusters that are running or terminated.

To view information about a cluster using the Amazon EMR CLI

You can view information about a cluster using the --describe parameter with the cluster ID.

Note

The Amazon EMR CLI is no longer under feature development. Customers are encouraged to use the Amazon EMR commands in the AWS CLI instead.

  • Use the --describe parameter with a valid cluster ID.

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --describe --jobflow JobFlowID
    • Windows users:

      ruby elastic-mapreduce --describe --jobflow JobFlowID

The response looks similar to the following:

{
  "JobFlows": [
    {
      "Name": "Development Job Flow (requires manual termination)",
      "LogUri": "s3n:\/\/AKIAIOSFODNN7EXAMPLE\/FileName\/",
      "ExecutionStatusDetail": {
        "StartDateTime": null,
        "EndDateTime": null,
        "LastStateChangeReason": "Starting instances",
        "CreationDateTime": DateTimeStamp,
        "State": "STARTING",
        "ReadyDateTime": null
      },
      "Steps": [],
      "Instances": {
        "MasterInstanceId": null,
        "Ec2KeyName": "KeyName",
        "NormalizedInstanceHours": 0,
        "InstanceCount": 5,
        "Placement": {
          "AvailabilityZone": "us-east-1a"
        },
        "SlaveInstanceType": "m1.small",
        "HadoopVersion": "0.20",
        "MasterPublicDnsName": null,
        "KeepJobFlowAliveWhenNoSteps": true,
        "InstanceGroups": [
          {
            "StartDateTime": null,
            "SpotPrice": null,
            "Name": "Master Instance Group",
            "InstanceRole": "MASTER",
            "EndDateTime": null,
            "LastStateChangeReason": "",
            "CreationDateTime": DateTimeStamp,
            "LaunchGroup": null,
            "InstanceGroupId": "InstanceGroupID",
            "State": "PROVISIONING",
            "Market": "ON_DEMAND",
            "ReadyDateTime": null,
            "InstanceType": "m1.small",
            "InstanceRunningCount": 0,
            "InstanceRequestCount": 1
          },
          {
            "StartDateTime": null,
            "SpotPrice": null,
            "Name": "Task Instance Group",
            "InstanceRole": "TASK",
            "EndDateTime": null,
            "LastStateChangeReason": "",
            "CreationDateTime": DateTimeStamp,
            "LaunchGroup": null,
            "InstanceGroupId": "InstanceGroupID",
            "State": "PROVISIONING",
            "Market": "ON_DEMAND",
            "ReadyDateTime": null,
            "InstanceType": "m1.small",
            "InstanceRunningCount": 0,
            "InstanceRequestCount": 2
          },
          {
            "StartDateTime": null,
            "SpotPrice": null,
            "Name": "Core Instance Group",
            "InstanceRole": "CORE",
            "EndDateTime": null,
            "LastStateChangeReason": "",
            "CreationDateTime": DateTimeStamp,
            "LaunchGroup": null,
            "InstanceGroupId": "InstanceGroupID",
            "State": "PROVISIONING",
            "Market": "ON_DEMAND",
            "ReadyDateTime": null,
            "InstanceType": "m1.small",
            "InstanceRunningCount": 0,
            "InstanceRequestCount": 2
          }
        ],
        "MasterInstanceType": "m1.small"
      },
      "bootstrapActions": [],
      "JobFlowId": "JobFlowID"
    }
  ]
}

For more information about the input parameters unique to DescribeJobFlows, see DescribeJobFlows.