Amazon Elastic MapReduce
Developer Guide (API Version 2009-03-31)
« PreviousNext »
View the PDF for this guide.Go to the AWS Discussion Forum for this product.Go to the Kindle Store to download this guide in Kindle format.Did this page help you?  Yes | No |  Tell us about it...

View Cluster Details

After you start a cluster, you can monitor its status and retrieve extended information about its execution. This section describes the methods used to view the details of Amazon EMR clusters. You can view clusters in any state.

This procedure explains how to view the details of a cluster using the Amazon EMR console.

To view the details of a cluster using the console

  1. Sign in to the AWS Management Console and open the Amazon Elastic MapReduce console at https://console.aws.amazon.com/elasticmapreduce/vnext/.

  2. Select the cluster to view.

    The Summary pane appears, providing detailed information about the selected cluster.

To view cluster details from the CLI, use the --list parameter to list clusters. This section presents some of these variations.

To list clusters created in the last two days using the CLI

  • Use the --list parameter with no additional arguments to display clusters created during the last two days as follows:

    In the directory where you installed the Amazon EMR CLI, run the following from the command line. For more information, see the Command Line Interface Reference for Amazon EMR.

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --list
    • Windows users:

      ruby elastic-mapreduce --list

The response is similar to the following:

j-1YE2DN7RXJBWU   FAILED      Example Job Flow
                  CANCELLED   Custom Jar
j-3GJ4FRRNKGY97   COMPLETED   ec2-67-202-3-73.compute-1.amazonaws.com   Example cluster
j-5XXFIQS8PFNW    COMPLETED   ec2-67-202-51-30.compute-1.amazonaws.com  demo 3/24 s1
                  COMPLETED   Custom Jar 

The example response shows that three clusters were created in the last two days. The indented lines are the steps of the cluster. The information for a cluster is in the following order: the cluster ID, the cluster state, the DNS name of the master node, and the cluster name. The information for a cluster step is in the following order: step state, and step name.

If no clusters were created in the previous two days, this command produces no output.

To list active clusters

  • Use the --list and --active parameters as follows:

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce  --list --active
    • Windows users:

      ruby elastic-mapreduce  --list --active

The response lists clusters that are in the state of STARTING, RUNNING, or SHUTTING_DOWN.

To list only running or terminated clusters

  • Use the --state parameter as follows:

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --list --state RUNNING  --state TERMINATED
    • Windows users:

      ruby elastic-mapreduce --list --state RUNNING  --state TERMINATED

The response lists clusters that are running or terminated.

You can get information about a cluster using the --describe parameter and specifying a cluster ID.

To retrieve information about a cluster

  • Use the --describe parameter with a valid cluster ID.

    • Linux, UNIX, and Mac OS X users:

      ./elastic-mapreduce --describe --jobflow JobFlowID
    • Windows users:

      ruby elastic-mapreduce --describe --jobflow JobFlowID

The response looks similar to the following:

{
  "JobFlows": [
    {
      "Name": "Development Job Flow (requires manual termination)",
      "LogUri": "s3n:\/\/AKIAIOSFODNN7EXAMPLE\/FileName\/",
      "ExecutionStatusDetail": {
        "StartDateTime": null,
        "EndDateTime": null,
        "LastStateChangeReason": "Starting instances",
        "CreationDateTime": DateTimeStamp,
        "State": "STARTING",
        "ReadyDateTime": null
      },
      "Steps": [],
      "Instances": {
        "MasterInstanceId": null,
        "Ec2KeyName": "KeyName",
        "NormalizedInstanceHours": 0,
        "InstanceCount": 5,
        "Placement": {
          "AvailabilityZone": "us-east-1a"
        },
        "SlaveInstanceType": "m1.small",
        "HadoopVersion": "0.20",
        "MasterPublicDnsName": null,
        "KeepJobFlowAliveWhenNoSteps": true,
        "InstanceGroups": [
          {
            "StartDateTime": null,
            "SpotPrice": null,
            "Name": "Master Instance Group",
            "InstanceRole": "MASTER",
            "EndDateTime": null,
            "LastStateChangeReason": "",
            "CreationDateTime": DateTimeStamp,
            "LaunchGroup": null,
            "InstanceGroupId": "InstanceGroupID",
            "State": "PROVISIONING",
            "Market": "ON_DEMAND",
            "ReadyDateTime": null,
            "InstanceType": "m1.small",
            "InstanceRunningCount": 0,
            "InstanceRequestCount": 1
          },
          {
            "StartDateTime": null,
            "SpotPrice": null,
            "Name": "Task Instance Group",
            "InstanceRole": "TASK",
            "EndDateTime": null,
            "LastStateChangeReason": "",
            "CreationDateTime": DateTimeStamp,
            "LaunchGroup": null,
            "InstanceGroupId": "InstanceGroupID",
            "State": "PROVISIONING",
            "Market": "ON_DEMAND",
            "ReadyDateTime": null,
            "InstanceType": "m1.small",
            "InstanceRunningCount": 0,
            "InstanceRequestCount": 2
          },
          {
            "StartDateTime": null,
            "SpotPrice": null,
            "Name": "Core Instance Group",
            "InstanceRole": "CORE",
            "EndDateTime": null,
            "LastStateChangeReason": "",
            "CreationDateTime": DateTimeStamp,
            "LaunchGroup": null,
            "InstanceGroupId": "InstanceGroupID",
            "State": "PROVISIONING",
            "Market": "ON_DEMAND",
            "ReadyDateTime": null,
            "InstanceType": "m1.small",
            "InstanceRunningCount": 0,
            "InstanceRequestCount": 2
          }
        ],
        "MasterInstanceType": "m1.small"
      },
      "bootstrapActions": [],
      "JobFlowId": "JobFlowID"
    }
  ]
}

Example using the API

The DescribeJobFlows operation in the Amazon EMR API returns details about specified clusters. You specify a cluster by the cluster ID, creation date, or state. Amazon EMR returns descriptions of clusters that are up to two months old. Specifying an older date returns an error. If you do not specify a CreatedAfter value, Amazon EMR uses the default of two months.


To return information about a cluster identified by its cluster ID

  • Issue a request similar to the following, replacing JobFlowID, AccessKeyID, and CalculatedValue with the values required for your cluster.

    https://elasticmapreduce.amazonaws.com?
    JobFlowIds.member.1=JobFlowID&
    Operation=DescribeJobFlows&
    AWSAccessKeyId=AccessKeyID&
    SignatureVersion=2&
    SignatureMethod=HmacSHA256&
    Timestamp=2009-01-28T21%3A49%3A59.000Z&
    Signature=CalculatedValue

    For more information about the input parameters unique to DescribeJobFlows, see DescribeJobFlows.

To return information about a clusters in a specific state

  • Issue a request similar to the following, replacing COMPLETED, AccessKeyID and CalculatedValue with the values required for your clusters.

    https://elasticmapreduce.amazonaws.com?
    JobFlowStates=COMPLETED&
    Operation=DescribeJobFlows&
    AWSAccessKeyId=AccessKeyID&
    SignatureVersion=2&
    SignatureMethod=HmacSHA256&
    Timestamp=2009-01-28T21%3A49%3A59.000Z&
    Signature=CalculatedValue

    For more information about the input parameters unique to DescribeJobFlows, see DescribeJobFlows.