Amazon EMR
Amazon EMR Release Guide

Customizing Cluster and Application Configuration With Earlier AMI Versions of Amazon EMR

Amazon EMR release version 4.0.0 introduced a simplified method of configuring applications using configuration classifications. For more information, see Configuring Applications. When using an AMI version, you configure applications using bootstrap actions along with arguments that you pass. For example, the configure-hadoop and configure-daemons bootstrap actions set Hadoop and YARN–specific environment properties like --namenode-heap-size. In more recent versions, these are configured using the hadoop-env and yarn-env configuration classifications. For bootstrap actions that perform common configurations, see the emr-bootstrap-actions repository on Github.

The following tables map bootstrap actions to configuration classifications in more recent Amazon EMR release versions.

Hadoop

Affected application file name AMI version bootstrap action Configuration classification
core-site.xml configure-hadoop -c core-site
log4j.properties configure-hadoop -l hadoop-log4j
hdfs-site.xml configure-hadoop -s hdfs-site
n/a n/a hdfs-encryption-zones
mapred-site.xml configure-hadoop -m mapred-site
yarn-site.xml configure-hadoop -y yarn-site
httpfs-site.xml configure-hadoop -t httpfs-site
capacity-scheduler.xml configure-hadoop -z capacity-scheduler
yarn-env.sh configure-daemons --resourcemanager-opts yarn-env

Hive

Affected application file name AMI version bootstrap action Configuration classification
hive-env.sh n/a hive-env
hive-site.xml hive-script --install-hive-site ${MY_HIVE_SITE_FILE} hive-site
hive-exec-log4j.properties n/a hive-exec-log4j
hive-log4j.properties n/a hive-log4j

EMRFS

Affected application file name AMI version bootstrap action Configuration classification
emrfs-site.xml configure-hadoop -e emrfs-site
n/a s3get -s s3://custom-provider.jar -d /usr/share/aws/emr/auxlib/ emrfs-site (with new setting fs.s3.cse.encryptionMaterialsProvider.uri)

For a list of all classifications, see Configuring Applications.

Application Environment Variables

When using an AMI version, a hadoop-user-env.sh script is used along with the configure-daemons bootstrap action to configure the Hadoop environment. The script includes the following actions:

#!/bin/bash export HADOOP_USER_CLASSPATH_FIRST=true; echo "HADOOP_CLASSPATH=/path/to/my.jar" >> /home/hadoop/conf/hadoop-user-env.sh

In Amazon EMR release 4.x, you do the same using the hadoop-env configuration classification, as shown in the following example:

[       {          "Classification":"hadoop-env",          "Properties":{          },          "Configurations":[             {                "Classification":"export",                "Properties":{                   "HADOOP_USER_CLASSPATH_FIRST":"true",                   "HADOOP_CLASSPATH":"/path/to/my.jar"                }             }          ]       }    ]

As another example, using configure-daemons and passing --namenode-heap-size=2048 and --namenode-opts=-XX:GCTimeRatio=19 is equivalent to the following configuration classifications.

[       {          "Classification":"hadoop-env",          "Properties":{          },          "Configurations":[             {                "Classification":"export",                "Properties":{                   "HADOOP_DATANODE_HEAPSIZE": "2048", "HADOOP_NAMENODE_OPTS": "-XX:GCTimeRatio=19"                }             }          ]       }    ]

Other application environment variables are no longer defined in /home/hadoop/.bashrc. Instead, they are primarily set in /etc/default files per component or application, such as /etc/default/hadoop. Wrapper scripts in /usr/bin/ installed by application RPMs may also set additional environment variables before involving the actual bin script.

Service Ports

When using an AMI version, some services use custom ports.

Changes in Port Settings

Setting AMI Version 3.x Open-source default
fs.default.name hdfs://emrDeterminedIP:9000 default (hdfs://emrDeterminedIP:8020)
dfs.datanode.address 0.0.0.0:9200 default (0.0.0.0:50010)
dfs.datanode.http.address 0.0.0.0:9102 default (0.0.0.0:50075)
dfs.datanode.https.address 0.0.0.0:9402 default (0.0.0.0:50475)
dfs.datanode.ipc.address 0.0.0.0:9201 default (0.0.0.0:50020)
dfs.http.address 0.0.0.0:9101 default (0.0.0.0:50070)
dfs.https.address 0.0.0.0:9202 default (0.0.0.0:50470)
dfs.secondary.http.address 0.0.0.0:9104 default (0.0.0.0:50090)
yarn.nodemanager.address 0.0.0.0:9103 default (${yarn.nodemanager.hostname}:0)
yarn.nodemanager.localizer.address 0.0.0.0:9033 default (${yarn.nodemanager.hostname}:8040)
yarn.nodemanager.webapp.address 0.0.0.0:9035 default (${yarn.nodemanager.hostname}:8042)
yarn.resourcemanager.address emrDeterminedIP:9022 default (${yarn.resourcemanager.hostname}:8032)
yarn.resourcemanager.admin.address emrDeterminedIP:9025 default (${yarn.resourcemanager.hostname}:8033)
yarn.resourcemanager.resource-tracker.address emrDeterminedIP:9023 default (${yarn.resourcemanager.hostname}:8031)
yarn.resourcemanager.scheduler.address emrDeterminedIP:9024 default (${yarn.resourcemanager.hostname}:8030)
yarn.resourcemanager.webapp.address 0.0.0.0:9026 default (${yarn.resourcemanager.hostname}:8088)
yarn.web-proxy.address emrDeterminedIP:9046 default (no-value)
yarn.resourcemanager.hostname 0.0.0.0 (default) emrDeterminedIP

Note

The emrDeterminedIP is an IP address that is generated by Amazon EMR.

Users

When using an AMI version, the user hadoop runs all processes and owns all files. In Amazon EMR release version 4.0.0 and later, users exist at the application and component level.

Installation Sequence, Installed Artifacts, and Log File Locations

When using an AMI version, application artifacts and their configuration directories are installed in the /home/hadoop/application directory. For example, if you installed Hive, the directory would be /home/hadoop/hive. In Amazon EMR release 4.0.0 and later, application artifacts are installed in the /usr/lib/application directory. When using an AMI version, log files are found in various places. The table below lists locations.

Changes in Log Locations on Amazon S3

Daemon or Application Directory location
instance-state node/instance-id/instance-state/
hadoop-hdfs-namenode daemons/instance-id/hadoop-hadoop-namenode.log
hadoop-hdfs-datanode daemons/instance-id/hadoop-hadoop-datanode.log
hadoop-yarn (ResourceManager) daemons/instance-id/yarn-hadoop-resourcemanager
hadoop-yarn (Proxy Server) daemons/instance-id/yarn-hadoop-proxyserver
mapred-historyserver daemons/instance-id/
httpfs daemons/instance-id/httpfs.log
hive-server node/instance-id/hive-server/hive-server.log
hive-metastore node/instance-id/apps/hive.log
Hive CLI node/instance-id/apps/hive.log
YARN applications user logs and container logs task-attempts/
Mahout N/A
Pig N/A
spark-historyserver N/A
mapreduce job history files jobs/

Command Runner

When using an AMI version, many scripts or programs, like /home/hadoop/contrib/streaming/hadoop-streaming.jar, are not placed on the shell login path environment, so you need to specify the full path when you use a jar file such as command-runner.jar or script-runner.jar to execute the scripts. The command-runner.jar is located on the AMI so there is no need to know a full URI as was the case with script-runner.jar.

Replication Factor

The replication factor lets you configure when to start a Hadoop JVM. You can start a new Hadoop JVM for every task, which provides better task isolation, or you can share JVMs between tasks, providing lower framework overhead. If you are processing many small files, it makes sense to reuse the JVM many times to amortize the cost of start-up. However, if each task takes a long time or processes a large amount of data, then you might choose to not reuse the JVM to ensure that all memory is freed for subsequent tasks. When using an AMI version, you can customize the replication factor using the configure-hadoop bootstrap action to set the mapred.job.reuse.jvm.num.tasks property.

The following example demonstrates setting the JVM reuse factor for infinite JVM reuse.

Note

Linux line continuation characters (\) are included for readability. They can be removed or used in Linux commands. For Windows, remove them or replace with a caret (^).

aws emr create-cluster --name "Test cluster" --ami-version 3.11.0 \ --applications Name=Hue Name=Hive Name=Pig \ --use-default-roles --ec2-attributes KeyName=myKey \ --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m3.xlarge \ InstanceGroupType=CORE,InstanceCount=2,InstanceType=m3.xlarge \ --bootstrap-actions Path=s3://elasticmapreduce/bootstrap-actions/configure-hadoop,\ Name="Configuring infinite JVM reuse",Args=["-m","mapred.job.reuse.jvm.num.tasks=-1"]