What is Amazon EMR?
With Amazon EMR (Amazon EMR) you can analyze and process vast amounts of data. It does this by distributing the computational work across a cluster of virtual servers running in the Amazon cloud. The cluster is managed using an open-source framework called Hadoop.
Hadoop uses a distributed processing architecture called MapReduce in which a task is mapped to a set of servers for processing. The results of the computation performed by those servers is then reduced down to a single output set. One node, designated as the master node, controls the distribution of tasks. The following diagram shows a Hadoop cluster with the master node directing a group of slave nodes which process the data.
Amazon EMR has made enhancements to Hadoop and other open-source applications to work seamlessly with AWS. For example, Hadoop clusters running on Amazon EMR use EC2 instances as virtual Linux servers for the master and slave nodes, Amazon S3 for bulk storage of input and output data, and CloudWatch to monitor cluster performance and raise alarms. You can also move data into and out of DynamoDB using Amazon EMR and Hive. All of this is orchestrated by Amazon EMR control software that launches and manages the Hadoop cluster. This process is called an Amazon EMR cluster.
The following diagram illustrates how Amazon EMR interacts with other AWS services.
Open-source projects that run on top of the Hadoop architecture can also be run on Amazon EMR. The most popular applications, such as Hive, Pig, HBase, DistCp, and Ganglia, are already integrated with Amazon EMR.
By running Hadoop on Amazon EMR you get the benefits of the cloud:
The ability to provision clusters of virtual servers within minutes.
You can scale the number of virtual servers in your cluster to manage your computation needs, and only pay for what you use.
Integration with other AWS services.
The following related resources can help you as you work with this service.
AWS Training and Courses – Links to role-based and specialty courses as well as self-paced labs to help sharpen your AWS skills and gain practical experience.
AWS Developer Tools – Links to developer tools and resources that provide documentation, code samples, release notes, and other information to help you build innovative applications with AWS.
AWS Support Center – The hub for creating and managing your AWS Support cases. Also includes links to other helpful resources, such as forums, technical FAQs, service health status, and AWS Trusted Advisor.
AWS Support – The primary web page for information about AWS Support, a one-on-one, fast-response support channel to help you build and run applications in the cloud.
Contact Us – A central contact point for inquiries concerning AWS billing, account, events, abuse, and other issues.
AWS Site Terms – Detailed information about our copyright and trademark; your account, license, and site access; and other topics.
AWS Big Data BlogThe AWS big data blog contains technical articles designed to help you collect, store, clean, process, and visualize big data.
Amazon EMR API Reference – Contains a technical description of all Amazon EMR APIs.
Getting Started Analyzing Big Data with AWS – This tutorial explains how to use Amazon EMR and Apache Hive to analyze web server log files and query them for information without writing any code at all.
Amazon EMR Technical FAQ – Covers the top questions developers have asked about this product.
Amazon EMR Release Notes – Gives a high-level overview of the current release, and notes any new features, corrections, and known issues.
Amazon EMR Articles and Tutorials – A list of articles, tutorials, and videos about Amazon EMR. Topics include tutorials that walk you through using Amazon EMR to solve a specific business problem and using third-party applications with Amazon EMR.
Amazon EMR Forum – A community-based forum for developers to discuss technical questions related to Amazon EMR.
Learn More About Hadoop and AWS Services Used with Amazon EMR:
Hadoop. For more information, go to http://hadoop.apache.org/core/.
Amazon Elastic Compute Cloud (Amazon EC2), Amazon Simple Storage Service (Amazon S3), and CloudWatch. For more information, see the Amazon EC2 User Guide for Linux Instances, the Amazon Simple Storage Service Developer Guide, Amazon SimpleDB Developer Guide and the Amazon CloudWatch Developer Guide, respectively.