Amazon Elastic MapReduce
Developer Guide (API Version 2009-03-31)
« PreviousNext »
View the PDF for this guide.Go to the AWS Discussion Forum for this product.Go to the Kindle Store to download this guide in Kindle format.Did this page help you?  Yes | No |  Tell us about it...

Monitor Performance with Ganglia

The Ganglia open source project is a scalable, distributed system designed to monitor clusters and grids while minimizing the impact on their performance. When you enable Ganglia on your cluster, you can generate reports and view the performance of the cluster as a whole, as well as inspect the performance of individual node instances. For more information about the Ganglia open-source project, go to http://ganglia.info/.

Add Ganglia to a Cluster

To add Ganglia to a cluster using the console

  1. Sign in to the AWS Management Console and open the Amazon Elastic MapReduce console at https://console.aws.amazon.com/elasticmapreduce/vnext/.

  2. Click Create Cluster.

    Create Cluster
  3. Under the Additional Applications list, choose Ganglia and click Configure and add.

  4. Proceed to create the cluster as described in Plan an Amazon EMR Cluster.

To add a Ganglia bootstrap action using the CLI

  • When you create a new cluster using the CLI, specify the Ganglia bootstrap action by adding the following parameter to your cluster call:

    --bootstrap-action s3://elasticmapreduce/bootstrap-actions/install-ganglia
    

The following command illustrates the use of the bootstrap-action parameter when starting a new cluster. In this example, you start the Word Count sample cluster provided by Amazon EMR and launch three instances.

In the directory where you installed the Amazon EMR CLI, run the following from the command line. For more information, see the Command Line Interface Reference for Amazon EMR.

Note

The Hadoop streaming syntax is different between Hadoop 1.x and Hadoop 2.x.

For Hadoop 2.x, use the following command:

  • Linux, UNIX, and Mac OS X users:

    ./elastic-mapreduce --create --alive --ami-version 3.0.3 --instance-type m1.xlarge \
    --num-instances 3 --stream --arg "-files" --arg "s3://elasticmapreduce/samples/wordcount/wordSplitter.py" \
    --bootstrap-action s3://elasticmapreduce/bootstrap-actions/install-ganglia --input s3://elasticmapreduce/samples/wordcount/input \
    --output s3://myawsbucket/output/2014-01-16 --mapper wordSplitter.py --reducer aggregate
  • Windows users:

    ruby elastic-mapreduce --create --alive --ami-version 3.0.3 --instance-type m1.xlarge --num-instances 3 --stream --arg "-files" --arg "s3://elasticmapreduce/samples/wordcount/wordSplitter.py" --bootstrap-action s3://elasticmapreduce/bootstrap-actions/install-ganglia --input s3://elasticmapreduce/samples/wordcount/input --output s3://myawsbucket/output/2014-01-16 --mapper wordSplitter.py --reducer aggregate 

For Hadoop 1.x, use the following command:

  • Linux, UNIX, and Mac OS X users:

    ./elastic-mapreduce --create --alive --instance-type m1.xlarge --num-instances 3 \
    --bootstrap-action s3://elasticmapreduce/bootstrap-actions/install-ganglia --stream \
    --input s3://elasticmapreduce/samples/wordcount/input \
    --output s3://myawsbucket/output/2014-01-16 \
    --mapper s3://elasticmapreduce/samples/wordcount/wordSplitter.py --reducer aggregate
  • Windows users:

    ruby elastic-mapreduce --create --alive --instance-type m1.xlarge --num-instances 3 --bootstrap-action s3://elasticmapreduce/bootstrap-actions/install-ganglia --stream --input s3://elasticmapreduce/samples/wordcount/input --output s3://myawsbucket/output/2014-01-16 --mapper s3://elasticmapreduce/samples/wordcount/wordSplitter.py --reducer aggregate

View Ganglia Metrics

Ganglia provides a web-based user interface that you can use to view the metrics Ganglia collects. When you run Ganglia on Amazon EMR, the web interface runs on the master node and can be viewed using port forwarding, also known as creating an SSH tunnel. For more information about viewing web interfaces on Amazon EMR, see Web Interfaces Hosted on the Master Node.

To view the Ganglia web interface

  1. Use SSH to tunnel into the master node and create a secure connection. For information about how to create an SSH tunnel to the master node, see Open an SSH Tunnel to the Master Node.

  2. Install a web browser with a proxy tool, such as the FoxyProxy plug-in for Firefox, to create a SOCKS proxy for domains of the type *ec2*.amazonaws.com*. For more information, see Configure FoxyProxy to View Websites Hosted on the Master Node.

  3. With the proxy set and the SSH connection open, you can view the Ganglia UI by opening a browser window with http://master-public-dns-name/ganglia/, where master-public-dns-name is the public DNS address of the master server in the Amazon EMR cluster. For information about how to locate the public DNS name of a master node, see To locate the public DNS name of the master node using the Amazon EMR console.

Ganglia Reports

When you open the Ganglia web reports in a browser, you see an overview of the cluster’s performance, with graphs detailing the load, memory usage, CPU utilization, and network traffic of the cluster. Below the cluster statistics are graphs for each individual server in the cluster. In the preceding cluster creation example, we launched three instances, so in the following reports there are three instance charts showing the cluster data.

Ganglia cluster report

The default graph for the node instances is Load, but you can use the Metric drop-down list to change the statistic displayed in the node-instance graphs.

Metric drop-down list

You can drill down into the full set of statistics for a given instance by selecting the node from the drop-down list or by clicking the corresponding node-instance chart.

Node drop-down list

This opens the Host Overview for the node.

Host overview

If you scroll down, you can view charts of the full range of statistics collected on the instance.

Instance statistics

Hadoop Metrics in Ganglia

Ganglia reports Hadoop metrics for each node instance. The various types of metrics are prefixed by category: distributed file system (dfs.*), Java virtual machine (jvm.*), MapReduce (mapred.*), and remote procedure calls (rpc.*). You can view a complete list of these metrics by clicking the Gmetrics link, on the Host Overview page.