The AWS Documentation website is getting a new look!
Try it now and let us know what you think. Switch to the new look >>

You can return to the original look by selecting English in the language selector above.

Amazon EMR 5.x Release Versions

Each tab below lists application versions, release notes, component versions, and configuration classifications available in each Amazon EMR 5.x release version.

For a comprehensive diagram of application versions in every release, see Application Versions in Amazon EMR 5.x Releases (PNG).

When you launch a cluster, you can choose from multiple release versions of Amazon EMR. This allows you to test and use application versions that fit your compatibility requirements. You specify the release version using the release label. Release labels are in the form emr-x.x.x. For example, emr-5.27.0.

New Amazon EMR release versions are made available in different regions over a period of several days, beginning with the first region on the initial release date. The latest release version may not be available in your region during this period.

5.27.05.26.05.25.05.24.x5.23.05.22.05.21.x5.20.05.19.05.18.05.17.x5.16.05.15.05.14.x5.13.05.12.x5.11.x5.10.05.9.05.8.x5.7.05.6.05.5.x5.4.05.3.x5.2.x5.1.05.0.x

5.27.0

Amazon EMR Release 5.27.0

5.27.0 Application Versions

The following applications are supported in this release: Flink, Ganglia, Hadoop, HBase, HCatalog, Hive, Hue, JupyterHub, Livy, Mahout, MXNet, Oozie, Phoenix, Pig, Presto, Spark, Sqoop, TensorFlow, Tez, Zeppelin, and ZooKeeper.

The diagram below depicts the application versions available in this release of Amazon EMR and the application versions in the preceding four Amazon EMR releases.

For a comprehensive history of application versions for each release of Amazon EMR, see the following diagrams:

5.27.0 Release Notes

The following release notes include information for Amazon EMR release version 5.27.0. Changes are relative to 5.26.0.

Initial release date: Sep 23, 2019

Upgrades

AWS SDK for Java 1.11.615
Flink 1.8.1
JupyterHub 1.0.0
Spark 2.4.4
Tensorflow 1.14.0
Connectors and drivers:
- DynamoDB Connector 4.12.0

Changes, Enhancements, and Resolved Issues

EMR cluster with multiple master nodes
- You can install and run Flink on an EMR cluster with multiple master nodes. For more information, see Supported Applications and Features.
- You can configure HDFS transparent encryption on an EMR cluster with multiple master nodes. For more information, see HDFS Transparent Encryption on EMR Clusters with Multiple Master Nodes.
- You can now modify the configuration of applications running on an EMR cluster with multiple master nodes. For more information, see Supplying a Configuration for an Instance Group in a Running Cluster.
Amazon EMR-DynamoDB Connector
- Amazon EMR-DynamoDB Connector now supports the following DynamoDB data types: boolean, list, map, item, null. For more information, see Set Up a Hive Table to Run Hive Commands.

5.27.0 Component Versions

The components that Amazon EMR installs with this release are listed below. Some are installed as part of big-data application packages. Others are unique to Amazon EMR and installed for system processes and features. These typically start with emr or aws. Big-data application packages in the most recent Amazon EMR release are usually the latest version found in the community. We make community releases available in Amazon EMR as quickly as possible.

Some components need changes from community versions for Amazon EMR. These components have a version label in the form CommunityVersion-amzn-EmrVersion. For example, if a big-data community component named myapp-component of version 2.2 has been modified three times for inclusion in different Amazon EMR release versions, its release version is listed as 2.2-amzn-3.

Component	Version	Description
aws-sagemaker-spark-sdk	1.2.4	Amazon SageMaker Spark SDK
emr-ddb	4.12.0	Amazon DynamoDB connector for Hadoop ecosystem applications.
emr-goodies	2.11.0	Extra convenience libraries for the Hadoop ecosystem.
emr-kinesis	3.4.0	Amazon Kinesis connector for Hadoop ecosystem applications.
emr-s3-dist-cp	2.13.0	Distributed copy application optimized for Amazon S3.
emr-s3-select	1.3.0	EMR S3Select Connector
emrfs	2.36.0	Amazon S3 connector for Hadoop ecosystem applications.
flink-client	1.8.1	Apache Flink command line client scripts and applications.
ganglia-monitor	3.7.2	Embedded Ganglia agent for Hadoop ecosystem applications along with the Ganglia monitoring agent.
ganglia-metadata-collector	3.7.2	Ganglia metadata collector for aggregating metrics from Ganglia monitoring agents.
ganglia-web	3.7.1	Web application for viewing metrics collected by the Ganglia metadata collector.
hadoop-client	2.8.5-amzn-4	Hadoop command-line clients such as 'hdfs', 'hadoop', or 'yarn'.
hadoop-hdfs-datanode	2.8.5-amzn-4	HDFS node-level service for storing blocks.
hadoop-hdfs-library	2.8.5-amzn-4	HDFS command-line client and library
hadoop-hdfs-namenode	2.8.5-amzn-4	HDFS service for tracking file names and block locations.
hadoop-hdfs-journalnode	2.8.5-amzn-4	HDFS service for managing the Hadoop filesystem journal on HA clusters.
hadoop-httpfs-server	2.8.5-amzn-4	HTTP endpoint for HDFS operations.
hadoop-kms-server	2.8.5-amzn-4	Cryptographic key management server based on Hadoop's KeyProvider API.
hadoop-mapred	2.8.5-amzn-4	MapReduce execution engine libraries for running a MapReduce application.
hadoop-yarn-nodemanager	2.8.5-amzn-4	YARN service for managing containers on an individual node.
hadoop-yarn-resourcemanager	2.8.5-amzn-4	YARN service for allocating and managing cluster resources and distributed applications.
hadoop-yarn-timeline-server	2.8.5-amzn-4	Service for retrieving current and historical information for YARN applications.
hbase-hmaster	1.4.10	Service for an HBase cluster responsible for coordination of Regions and execution of administrative commands.
hbase-region-server	1.4.10	Service for serving one or more HBase regions.
hbase-client	1.4.10	HBase command-line client.
hbase-rest-server	1.4.10	Service providing a RESTful HTTP endpoint for HBase.
hbase-thrift-server	1.4.10	Service providing a Thrift endpoint to HBase.
hcatalog-client	2.3.5-amzn-1	The 'hcat' command line client for manipulating hcatalog-server.
hcatalog-server	2.3.5-amzn-1	Service providing HCatalog, a table and storage management layer for distributed applications.
hcatalog-webhcat-server	2.3.5-amzn-1	HTTP endpoint providing a REST interface to HCatalog.
hive-client	2.3.5-amzn-1	Hive command line client.
hive-hbase	2.3.5-amzn-1	Hive-hbase client.
hive-metastore-server	2.3.5-amzn-1	Service for accessing the Hive metastore, a semantic repository storing metadata for SQL on Hadoop operations.
hive-server2	2.3.5-amzn-1	Service for accepting Hive queries as web requests.
hue-server	4.4.0	Web application for analyzing data using Hadoop ecosystem applications
jupyterhub	1.0.0	Multi-user server for Jupyter notebooks
livy-server	0.6.0-incubating	REST interface for interacting with Apache Spark
nginx	1.12.1	nginx [engine x] is an HTTP and reverse proxy server
mahout-client	0.13.0	Library for machine learning.
mxnet	1.4.0	A flexible, scalable, and efficient library for deep learning.
mysql-server	5.5.54+	MySQL database server.
nvidia-cuda	9.2.88	Nvidia drivers and Cuda toolkit
oozie-client	5.1.0	Oozie command-line client.
oozie-server	5.1.0	Service for accepting Oozie workflow requests.
opencv	3.4.0	Open Source Computer Vision Library.
phoenix-library	4.14.2-HBase-1.4	The phoenix libraries for server and client
phoenix-query-server	4.14.2-HBase-1.4	A light weight server providing JDBC access as well as Protocol Buffers and JSON format access to the Avatica API
presto-coordinator	0.224	Service for accepting queries and managing query execution among presto-workers.
presto-worker	0.224	Service for executing pieces of a query.
pig-client	0.17.0	Pig command-line client.
r	3.4.1	The R Project for Statistical Computing
spark-client	2.4.4	Spark command-line clients.
spark-history-server	2.4.4	Web UI for viewing logged events for the lifetime of a completed Spark application.
spark-on-yarn	2.4.4	In-memory execution engine for YARN.
spark-yarn-slave	2.4.4	Apache Spark libraries needed by YARN slaves.
sqoop-client	1.4.7	Apache Sqoop command-line client.
tensorflow	1.14.0	TensorFlow open source software library for high performance numerical computation.
tez-on-yarn	0.9.2	The tez YARN application and libraries.
webserver	2.4.25+	Apache HTTP server.
zeppelin-server	0.8.1	Web-based notebook that enables interactive data analytics.
zookeeper-server	3.4.14	Centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
zookeeper-client	3.4.14	ZooKeeper command line client.

5.27.0 Configuration Classifications

Configuration classifications allow you to customize applications. These often correspond to a configuration XML file for the application, such as hive-site.xml. For more information, see Configuring Applications.

emr-5.27.0 Classifications

Classifications	Description
capacity-scheduler	Change values in Hadoop's capacity-scheduler.xml file.
container-log4j	Change values in Hadoop YARN's container-log4j.properties file.
core-site	Change values in Hadoop's core-site.xml file.
emrfs-site	Change EMRFS settings.
flink-conf	Change flink-conf.yaml settings.
flink-log4j	Change Flink log4j.properties settings.
flink-log4j-yarn-session	Change Flink log4j-yarn-session.properties settings.
flink-log4j-cli	Change Flink log4j-cli.properties settings.
hadoop-env	Change values in the Hadoop environment for all Hadoop components.
hadoop-log4j	Change values in Hadoop's log4j.properties file.
hadoop-ssl-server	Change hadoop ssl server configuration
hadoop-ssl-client	Change hadoop ssl client configuration
hbase	Amazon EMR-curated settings for Apache HBase.
hbase-env	Change values in HBase's environment.
hbase-log4j	Change values in HBase's hbase-log4j.properties file.
hbase-metrics	Change values in HBase's hadoop-metrics2-hbase.properties file.
hbase-policy	Change values in HBase's hbase-policy.xml file.
hbase-site	Change values in HBase's hbase-site.xml file.
hdfs-encryption-zones	Configure HDFS encryption zones.
hdfs-site	Change values in HDFS's hdfs-site.xml.
hcatalog-env	Change values in HCatalog's environment.
hcatalog-server-jndi	Change values in HCatalog's jndi.properties.
hcatalog-server-proto-hive-site	Change values in HCatalog's proto-hive-site.xml.
hcatalog-webhcat-env	Change values in HCatalog WebHCat's environment.
hcatalog-webhcat-log4j2	Change values in HCatalog WebHCat's log4j2.properties.
hcatalog-webhcat-site	Change values in HCatalog WebHCat's webhcat-site.xml file.
hive-beeline-log4j2	Change values in Hive's beeline-log4j2.properties file.
hive-parquet-logging	Change values in Hive's parquet-logging.properties file.
hive-env	Change values in the Hive environment.
hive-exec-log4j2	Change values in Hive's hive-exec-log4j2.properties file.
hive-llap-daemon-log4j2	Change values in Hive's llap-daemon-log4j2.properties file.
hive-log4j2	Change values in Hive's hive-log4j2.properties file.
hive-site	Change values in Hive's hive-site.xml file
hiveserver2-site	Change values in Hive Server2's hiveserver2-site.xml file
hue-ini	Change values in Hue's ini file
httpfs-env	Change values in the HTTPFS environment.
httpfs-site	Change values in Hadoop's httpfs-site.xml file.
hadoop-kms-acls	Change values in Hadoop's kms-acls.xml file.
hadoop-kms-env	Change values in the Hadoop KMS environment.
hadoop-kms-log4j	Change values in Hadoop's kms-log4j.properties file.
hadoop-kms-site	Change values in Hadoop's kms-site.xml file.
jupyter-notebook-conf	Change values in Jupyter Notebook's jupyter_notebook_config.py file.
jupyter-hub-conf	Change values in JupyterHubs's jupyterhub_config.py file.
jupyter-s3-conf	Configure Jupyter Notebook S3 persistence.
jupyter-sparkmagic-conf	Change values in Sparkmagic's config.json file.
livy-conf	Change values in Livy's livy.conf file.
livy-env	Change values in the Livy environment.
livy-log4j	Change Livy log4j.properties settings.
mapred-env	Change values in the MapReduce application's environment.
mapred-site	Change values in the MapReduce application's mapred-site.xml file.
oozie-env	Change values in Oozie's environment.
oozie-log4j	Change values in Oozie's oozie-log4j.properties file.
oozie-site	Change values in Oozie's oozie-site.xml file.
phoenix-hbase-metrics	Change values in Phoenix's hadoop-metrics2-hbase.properties file.
phoenix-hbase-site	Change values in Phoenix's hbase-site.xml file.
phoenix-log4j	Change values in Phoenix's log4j.properties file.
phoenix-metrics	Change values in Phoenix's hadoop-metrics2-phoenix.properties file.
pig-env	Change values in the Pig environment.
pig-properties	Change values in Pig's pig.properties file.
pig-log4j	Change values in Pig's log4j.properties file.
presto-log	Change values in Presto's log.properties file.
presto-config	Change values in Presto's config.properties file.
presto-password-authenticator	Change values in Presto's password-authenticator.properties file.
presto-env	Change values in Presto's presto-env.sh file.
presto-node	Change values in Presto's node.properties file.
presto-connector-blackhole	Change values in Presto's blackhole.properties file.
presto-connector-cassandra	Change values in Presto's cassandra.properties file.
presto-connector-hive	Change values in Presto's hive.properties file.
presto-connector-jmx	Change values in Presto's jmx.properties file.
presto-connector-kafka	Change values in Presto's kafka.properties file.
presto-connector-localfile	Change values in Presto's localfile.properties file.
presto-connector-memory	Change values in Presto's memory.properties file.
presto-connector-mongodb	Change values in Presto's mongodb.properties file.
presto-connector-mysql	Change values in Presto's mysql.properties file.
presto-connector-postgresql	Change values in Presto's postgresql.properties file.
presto-connector-raptor	Change values in Presto's raptor.properties file.
presto-connector-redis	Change values in Presto's redis.properties file.
presto-connector-redshift	Change values in Presto's redshift.properties file.
presto-connector-tpch	Change values in Presto's tpch.properties file.
presto-connector-tpcds	Change values in Presto's tpcds.properties file.
ranger-kms-dbks-site	Change values in dbks-site.xml file of Ranger KMS.
ranger-kms-site	Change values in ranger-kms-site.xml file of Ranger KMS.
ranger-kms-env	Change values in the Ranger KMS environment.
ranger-kms-log4j	Change values in kms-log4j.properties file of Ranger KMS.
ranger-kms-db-ca	Change values for CA file on S3 for MySQL SSL connection with Ranger KMS.
recordserver-env	Change values in the EMR RecordServer environment.
recordserver-conf	Change values in EMR RecordServer's erver.properties file.
recordserver-log4j	Change values in EMR RecordServer's log4j.properties file.
spark	Amazon EMR-curated settings for Apache Spark.
spark-defaults	Change values in Spark's spark-defaults.conf file.
spark-env	Change values in the Spark environment.
spark-hive-site	Change values in Spark's hive-site.xml file
spark-log4j	Change values in Spark's log4j.properties file.
spark-metrics	Change values in Spark's metrics.properties file.
sqoop-env	Change values in Sqoop's environment.
sqoop-oraoop-site	Change values in Sqoop OraOop's oraoop-site.xml file.
sqoop-site	Change values in Sqoop's sqoop-site.xml file.
tez-site	Change values in Tez's tez-site.xml file.
yarn-env	Change values in the YARN environment.
yarn-site	Change values in YARN's yarn-site.xml file.
zeppelin-env	Change values in the Zeppelin environment.
zookeeper-config	Change values in ZooKeeper's zoo.cfg file.
zookeeper-log4j	Change values in ZooKeeper's log4j.properties file.

5.26.0

Amazon EMR Release 5.26.0

5.26.0 Application Versions

The diagram below depicts the application versions available in this release of Amazon EMR and the application versions in the preceding four Amazon EMR releases.

For a comprehensive history of application versions for each release of Amazon EMR, see the following diagrams:

5.26.0 Release Notes

The following release notes include information for Amazon EMR release version 5.26.0. Changes are relative to 5.25.0.

Initial release date: Aug 8, 2019

Last updated date: Aug 19, 2019

Upgrades

AWS SDK for Java 1.11.595
HBase 1.4.10
Phoenix 4.14.2
Connectors and drivers:
- DynamoDB Connector 4.11.0
- MariaDB Connector 2.4.2
- Amazon Redshift JDBC Driver 1.2.32.1056

New Features

(Beta) With Amazon EMR 5.26.0, you can launch a cluster that integrates with Lake Formation. This integration provides fine-grained, column-level access to databases and tables in the AWS Glue Data Catalog. It also enables federated single sign-on to EMR Notebooks or Apache Zeppelin from an enterprise identity system. For more information, see Integrating Amazon EMR with AWS Lake Formation (Beta).
(Aug 19, 2019) Amazon EMR block public access is now available with all Amazon EMR releases that support security groups. Block public access is an account-wide setting applied to each AWS Region. Block public access prevents a cluster from launching when any security group associated with the cluster has a rule that allows inbound traffic from IPv4 0.0.0.0/0 or IPv6 ::/0 (public access) on a port, unless a port is specified as an exception. Port 22 is an exception by default. For more information, see Using Amazon EMR Block Public Access in the Amazon EMR Management Guide.

Changes, Enhancements, and Resolved Issues

EMR Notebooks
- With EMR 5.26.0 and later, EMR Notebooks supports notebook-scoped Python libraries in addition to the default Python libraries. You can install notebook-scoped libraries from within the notebook editor without having to re-create a cluster or re-attach a notebook to a cluster. Notebook-scoped libraries are created in a Python virtual environment, so they apply only to the current notebook session. This allows you to isolate notebook dependencies. For more information, see Using Notebook Scoped Libraries in the Amazon EMR Management Guide.
EMRFS
- You can enable an ETag verification feature (Beta) by setting fs.s3.consistent.metadata.etag.verification.enabled to true. With this feature, EMRFS uses Amazon S3 ETags to verify that objects being read are the latest available version. This feature is helpful for read-after-update use cases in which files on Amazon S3 are overwritten while retaining the same name. This ETag verification capability currently does not work with S3 Select. For more information, see Configure Consistent View.
Spark
- The following optimizations are now enabled by default: dynamic partition pruning, DISTINCT before INTERSECT, improvements in SQL plan statistics inference for JOIN followed by DISTINCT queries, flattening scalar subqueries, optimized join reorder, and bloom filter join. For more information, see Optimizing Spark Performance.
- Improved whole stage code generation for Sort Merge Join.
- Improved query fragment and subquery reuse.
- Improvements to pre-allocate executors on Spark start up.
- Bloom filter joins are no longer applied when the smaller side of the join includes a broadcast hint.
Tez
- Resolved an issue with Tez. Tez UI now works on an EMR cluster with multiple master nodes.

Known Issues

The improved whole stage code generation capabilities for Sort Merge Join can increase memory pressure when enabled. This optimization improves performance, but may result in job retries or failures if the spark.yarn.executor.memoryOverheadFactor is not tuned to provide enough memory. To disable this feature, set spark.sql.sortMergeJoinExec.extendedCodegen.enabled to false.

5.26.0 Component Versions

Component	Version	Description
aws-sagemaker-spark-sdk	1.2.4	Amazon SageMaker Spark SDK
emr-ddb	4.11.0	Amazon DynamoDB connector for Hadoop ecosystem applications.
emr-goodies	2.10.0	Extra convenience libraries for the Hadoop ecosystem.
emr-kinesis	3.4.0	Amazon Kinesis connector for Hadoop ecosystem applications.
emr-s3-dist-cp	2.12.0	Distributed copy application optimized for Amazon S3.
emr-s3-select	1.3.0	EMR S3Select Connector
emrfs	2.35.0	Amazon S3 connector for Hadoop ecosystem applications.
flink-client	1.8.0	Apache Flink command line client scripts and applications.
ganglia-monitor	3.7.2	Embedded Ganglia agent for Hadoop ecosystem applications along with the Ganglia monitoring agent.
ganglia-metadata-collector	3.7.2	Ganglia metadata collector for aggregating metrics from Ganglia monitoring agents.
ganglia-web	3.7.1	Web application for viewing metrics collected by the Ganglia metadata collector.
hadoop-client	2.8.5-amzn-4	Hadoop command-line clients such as 'hdfs', 'hadoop', or 'yarn'.
hadoop-hdfs-datanode	2.8.5-amzn-4	HDFS node-level service for storing blocks.
hadoop-hdfs-library	2.8.5-amzn-4	HDFS command-line client and library
hadoop-hdfs-namenode	2.8.5-amzn-4	HDFS service for tracking file names and block locations.
hadoop-hdfs-journalnode	2.8.5-amzn-4	HDFS service for managing the Hadoop filesystem journal on HA clusters.
hadoop-httpfs-server	2.8.5-amzn-4	HTTP endpoint for HDFS operations.
hadoop-kms-server	2.8.5-amzn-4	Cryptographic key management server based on Hadoop's KeyProvider API.
hadoop-mapred	2.8.5-amzn-4	MapReduce execution engine libraries for running a MapReduce application.
hadoop-yarn-nodemanager	2.8.5-amzn-4	YARN service for managing containers on an individual node.
hadoop-yarn-resourcemanager	2.8.5-amzn-4	YARN service for allocating and managing cluster resources and distributed applications.
hadoop-yarn-timeline-server	2.8.5-amzn-4	Service for retrieving current and historical information for YARN applications.
hbase-hmaster	1.4.10	Service for an HBase cluster responsible for coordination of Regions and execution of administrative commands.
hbase-region-server	1.4.10	Service for serving one or more HBase regions.
hbase-client	1.4.10	HBase command-line client.
hbase-rest-server	1.4.10	Service providing a RESTful HTTP endpoint for HBase.
hbase-thrift-server	1.4.10	Service providing a Thrift endpoint to HBase.
hcatalog-client	2.3.5-amzn-0	The 'hcat' command line client for manipulating hcatalog-server.
hcatalog-server	2.3.5-amzn-0	Service providing HCatalog, a table and storage management layer for distributed applications.
hcatalog-webhcat-server	2.3.5-amzn-0	HTTP endpoint providing a REST interface to HCatalog.
hive-client	2.3.5-amzn-0	Hive command line client.
hive-hbase	2.3.5-amzn-0	Hive-hbase client.
hive-metastore-server	2.3.5-amzn-0	Service for accessing the Hive metastore, a semantic repository storing metadata for SQL on Hadoop operations.
hive-server2	2.3.5-amzn-0	Service for accepting Hive queries as web requests.
hue-server	4.4.0	Web application for analyzing data using Hadoop ecosystem applications
jupyterhub	0.9.6	Multi-user server for Jupyter notebooks
livy-server	0.6.0-incubating	REST interface for interacting with Apache Spark
nginx	1.12.1	nginx [engine x] is an HTTP and reverse proxy server
mahout-client	0.13.0	Library for machine learning.
mxnet	1.4.0	A flexible, scalable, and efficient library for deep learning.
mysql-server	5.5.54+	MySQL database server.
nvidia-cuda	9.2.88	Nvidia drivers and Cuda toolkit
oozie-client	5.1.0	Oozie command-line client.
oozie-server	5.1.0	Service for accepting Oozie workflow requests.
opencv	3.4.0	Open Source Computer Vision Library.
phoenix-library	4.14.2-HBase-1.4	The phoenix libraries for server and client
phoenix-query-server	4.14.2-HBase-1.4	A light weight server providing JDBC access as well as Protocol Buffers and JSON format access to the Avatica API
presto-coordinator	0.220	Service for accepting queries and managing query execution among presto-workers.
presto-worker	0.220	Service for executing pieces of a query.
pig-client	0.17.0	Pig command-line client.
r	3.4.1	The R Project for Statistical Computing
spark-client	2.4.3	Spark command-line clients.
spark-history-server	2.4.3	Web UI for viewing logged events for the lifetime of a completed Spark application.
spark-on-yarn	2.4.3	In-memory execution engine for YARN.
spark-yarn-slave	2.4.3	Apache Spark libraries needed by YARN slaves.
sqoop-client	1.4.7	Apache Sqoop command-line client.
tensorflow	1.13.1	TensorFlow open source software library for high performance numerical computation.
tez-on-yarn	0.9.2	The tez YARN application and libraries.
webserver	2.4.25+	Apache HTTP server.
zeppelin-server	0.8.1	Web-based notebook that enables interactive data analytics.
zookeeper-server	3.4.14	Centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
zookeeper-client	3.4.14	ZooKeeper command line client.

5.26.0 Configuration Classifications

emr-5.26.0 Classifications

Classifications	Description
capacity-scheduler	Change values in Hadoop's capacity-scheduler.xml file.
container-log4j	Change values in Hadoop YARN's container-log4j.properties file.
core-site	Change values in Hadoop's core-site.xml file.
emrfs-site	Change EMRFS settings.
flink-conf	Change flink-conf.yaml settings.
flink-log4j	Change Flink log4j.properties settings.
flink-log4j-yarn-session	Change Flink log4j-yarn-session.properties settings.
flink-log4j-cli	Change Flink log4j-cli.properties settings.
hadoop-env	Change values in the Hadoop environment for all Hadoop components.
hadoop-log4j	Change values in Hadoop's log4j.properties file.
hadoop-ssl-server	Change hadoop ssl server configuration
hadoop-ssl-client	Change hadoop ssl client configuration
hbase	Amazon EMR-curated settings for Apache HBase.
hbase-env	Change values in HBase's environment.
hbase-log4j	Change values in HBase's hbase-log4j.properties file.
hbase-metrics	Change values in HBase's hadoop-metrics2-hbase.properties file.
hbase-policy	Change values in HBase's hbase-policy.xml file.
hbase-site	Change values in HBase's hbase-site.xml file.
hdfs-encryption-zones	Configure HDFS encryption zones.
hdfs-site	Change values in HDFS's hdfs-site.xml.
hcatalog-env	Change values in HCatalog's environment.
hcatalog-server-jndi	Change values in HCatalog's jndi.properties.
hcatalog-server-proto-hive-site	Change values in HCatalog's proto-hive-site.xml.
hcatalog-webhcat-env	Change values in HCatalog WebHCat's environment.
hcatalog-webhcat-log4j2	Change values in HCatalog WebHCat's log4j2.properties.
hcatalog-webhcat-site	Change values in HCatalog WebHCat's webhcat-site.xml file.
hive-beeline-log4j2	Change values in Hive's beeline-log4j2.properties file.
hive-parquet-logging	Change values in Hive's parquet-logging.properties file.
hive-env	Change values in the Hive environment.
hive-exec-log4j2	Change values in Hive's hive-exec-log4j2.properties file.
hive-llap-daemon-log4j2	Change values in Hive's llap-daemon-log4j2.properties file.
hive-log4j2	Change values in Hive's hive-log4j2.properties file.
hive-site	Change values in Hive's hive-site.xml file
hiveserver2-site	Change values in Hive Server2's hiveserver2-site.xml file
hue-ini	Change values in Hue's ini file
httpfs-env	Change values in the HTTPFS environment.
httpfs-site	Change values in Hadoop's httpfs-site.xml file.
hadoop-kms-acls	Change values in Hadoop's kms-acls.xml file.
hadoop-kms-env	Change values in the Hadoop KMS environment.
hadoop-kms-log4j	Change values in Hadoop's kms-log4j.properties file.
hadoop-kms-site	Change values in Hadoop's kms-site.xml file.
jupyter-notebook-conf	Change values in Jupyter Notebook's jupyter_notebook_config.py file.
jupyter-hub-conf	Change values in JupyterHubs's jupyterhub_config.py file.
jupyter-s3-conf	Configure Jupyter Notebook S3 persistence.
jupyter-sparkmagic-conf	Change values in Sparkmagic's config.json file.
livy-conf	Change values in Livy's livy.conf file.
livy-env	Change values in the Livy environment.
livy-log4j	Change Livy log4j.properties settings.
mapred-env	Change values in the MapReduce application's environment.
mapred-site	Change values in the MapReduce application's mapred-site.xml file.
oozie-env	Change values in Oozie's environment.
oozie-log4j	Change values in Oozie's oozie-log4j.properties file.
oozie-site	Change values in Oozie's oozie-site.xml file.
phoenix-hbase-metrics	Change values in Phoenix's hadoop-metrics2-hbase.properties file.
phoenix-hbase-site	Change values in Phoenix's hbase-site.xml file.
phoenix-log4j	Change values in Phoenix's log4j.properties file.
phoenix-metrics	Change values in Phoenix's hadoop-metrics2-phoenix.properties file.
pig-env	Change values in the Pig environment.
pig-properties	Change values in Pig's pig.properties file.
pig-log4j	Change values in Pig's log4j.properties file.
presto-log	Change values in Presto's log.properties file.
presto-config	Change values in Presto's config.properties file.
presto-password-authenticator	Change values in Presto's password-authenticator.properties file.
presto-env	Change values in Presto's presto-env.sh file.
presto-node	Change values in Presto's node.properties file.
presto-connector-blackhole	Change values in Presto's blackhole.properties file.
presto-connector-cassandra	Change values in Presto's cassandra.properties file.
presto-connector-hive	Change values in Presto's hive.properties file.
presto-connector-jmx	Change values in Presto's jmx.properties file.
presto-connector-kafka	Change values in Presto's kafka.properties file.
presto-connector-localfile	Change values in Presto's localfile.properties file.
presto-connector-memory	Change values in Presto's memory.properties file.
presto-connector-mongodb	Change values in Presto's mongodb.properties file.
presto-connector-mysql	Change values in Presto's mysql.properties file.
presto-connector-postgresql	Change values in Presto's postgresql.properties file.
presto-connector-raptor	Change values in Presto's raptor.properties file.
presto-connector-redis	Change values in Presto's redis.properties file.
presto-connector-redshift	Change values in Presto's redshift.properties file.
presto-connector-tpch	Change values in Presto's tpch.properties file.
presto-connector-tpcds	Change values in Presto's tpcds.properties file.
recordserver-env	Change values in the EMR RecordServer environment.
recordserver-conf	Change values in EMR RecordServer's erver.properties file.
recordserver-log4j	Change values in EMR RecordServer's log4j.properties file.
spark	Amazon EMR-curated settings for Apache Spark.
spark-defaults	Change values in Spark's spark-defaults.conf file.
spark-env	Change values in the Spark environment.
spark-hive-site	Change values in Spark's hive-site.xml file
spark-log4j	Change values in Spark's log4j.properties file.
spark-metrics	Change values in Spark's metrics.properties file.
sqoop-env	Change values in Sqoop's environment.
sqoop-oraoop-site	Change values in Sqoop OraOop's oraoop-site.xml file.
sqoop-site	Change values in Sqoop's sqoop-site.xml file.
tez-site	Change values in Tez's tez-site.xml file.
yarn-env	Change values in the YARN environment.
yarn-site	Change values in YARN's yarn-site.xml file.
zeppelin-env	Change values in the Zeppelin environment.
zookeeper-config	Change values in ZooKeeper's zoo.cfg file.
zookeeper-log4j	Change values in ZooKeeper's log4j.properties file.

5.25.0

Amazon EMR Release 5.25.0

5.25.0 Application Versions

The diagram below depicts the application versions available in this release of Amazon EMR and the application versions in the preceding four Amazon EMR releases.

For a comprehensive history of application versions for each release of Amazon EMR, see the following diagrams:

5.25.0 Release Notes

The following release notes include information for Amazon EMR release version 5.25.0. Changes are relative to 5.24.1.

Initial release date: July 17, 2019

Last updated date: August 5, 2019

Amazon EMR 5.25.0

Upgrades

AWS SDK for Java 1.11.566
Hive 2.3.5
Presto 0.220
Spark 2.4.3
TensorFlow 1.13.1
Tez 0.9.2
Zookeeper 3.4.14

Changes, Enhancements, and Resolved Issues

Spark
- Improved the performance of some joins by using Bloom filters to pre-filter inputs. The optimization is disabled by default and can be enabled by setting the Spark configuration parameter spark.sql.bloomFilterJoin.enabled to true.
- Improved the performance of grouping by string type columns.
- Improved the default Spark executor memory and cores configuration of R4 instance types for clusters without HBase installed.
- Resolved a previous issue with the dynamic partition pruning feature where the pruned table has to be on the left side of the join.
- Improved DISTINCT before INTERSECT optimization to apply to additional cases involving aliases.
- Improved SQL plan statistics inference for JOIN followed by DISTINCT queries. This improvement is disabled by default and can be enabled by setting the Spark configuration parameter spark.sql.statsImprovements.enabled to true. This optimization is required by the Distinct before Intersect feature and will be enabled automatically when spark.sql.optimizer.distinctBeforeIntersect.enabled is set to true.
- Optimized join order based on table size and filters. This optimization is disabled by default and can be enabled by setting the Spark configuration parameter spark.sql.optimizer.sizeBasedJoinReorder.enabled to true.
For more information, see Optimizing Spark Performance.
EMRFS
- The EMRFS setting, fs.s3.buckets.create.enabled, is now disabled by default. With testing, we found that disabling this setting improves performance and prevents unintentional creation of S3 buckets. If your application relies on this functionality, you can enable it by setting the property fs.s3.buckets.create.enabled to true in the emrfs-site configuration classification. For information, see Supplying a Configuration when Creating a Cluster.
Local Disk Encryption and S3 Encryption Improvements in Security Configurations (August 5, 2019)
- Separated Amazon S3 encryption settings from local disk encryption settings in security configuration setup.
- Added an option to enable EBS encryption with release 5.24.0 and later. Selecting this option encrypts the root device volume in addition to storage volumes. Previous versions required using a custom AMI to encrypt the root device volume.
- For more information, see Encryption Options in the Amazon EMR Management Guide.

5.25.0 Component Versions

Component	Version	Description
aws-sagemaker-spark-sdk	1.2.4	Amazon SageMaker Spark SDK
emr-ddb	4.10.0	Amazon DynamoDB connector for Hadoop ecosystem applications.
emr-goodies	2.9.0	Extra convenience libraries for the Hadoop ecosystem.
emr-kinesis	3.4.0	Amazon Kinesis connector for Hadoop ecosystem applications.
emr-s3-dist-cp	2.11.0	Distributed copy application optimized for Amazon S3.
emr-s3-select	1.3.0	EMR S3Select Connector
emrfs	2.34.0	Amazon S3 connector for Hadoop ecosystem applications.
flink-client	1.8.0	Apache Flink command line client scripts and applications.
ganglia-monitor	3.7.2	Embedded Ganglia agent for Hadoop ecosystem applications along with the Ganglia monitoring agent.
ganglia-metadata-collector	3.7.2	Ganglia metadata collector for aggregating metrics from Ganglia monitoring agents.
ganglia-web	3.7.1	Web application for viewing metrics collected by the Ganglia metadata collector.
hadoop-client	2.8.5-amzn-4	Hadoop command-line clients such as 'hdfs', 'hadoop', or 'yarn'.
hadoop-hdfs-datanode	2.8.5-amzn-4	HDFS node-level service for storing blocks.
hadoop-hdfs-library	2.8.5-amzn-4	HDFS command-line client and library
hadoop-hdfs-namenode	2.8.5-amzn-4	HDFS service for tracking file names and block locations.
hadoop-hdfs-journalnode	2.8.5-amzn-4	HDFS service for managing the Hadoop filesystem journal on HA clusters.
hadoop-httpfs-server	2.8.5-amzn-4	HTTP endpoint for HDFS operations.
hadoop-kms-server	2.8.5-amzn-4	Cryptographic key management server based on Hadoop's KeyProvider API.
hadoop-mapred	2.8.5-amzn-4	MapReduce execution engine libraries for running a MapReduce application.
hadoop-yarn-nodemanager	2.8.5-amzn-4	YARN service for managing containers on an individual node.
hadoop-yarn-resourcemanager	2.8.5-amzn-4	YARN service for allocating and managing cluster resources and distributed applications.
hadoop-yarn-timeline-server	2.8.5-amzn-4	Service for retrieving current and historical information for YARN applications.
hbase-hmaster	1.4.9	Service for an HBase cluster responsible for coordination of Regions and execution of administrative commands.
hbase-region-server	1.4.9	Service for serving one or more HBase regions.
hbase-client	1.4.9	HBase command-line client.
hbase-rest-server	1.4.9	Service providing a RESTful HTTP endpoint for HBase.
hbase-thrift-server	1.4.9	Service providing a Thrift endpoint to HBase.
hcatalog-client	2.3.5-amzn-0	The 'hcat' command line client for manipulating hcatalog-server.
hcatalog-server	2.3.5-amzn-0	Service providing HCatalog, a table and storage management layer for distributed applications.
hcatalog-webhcat-server	2.3.5-amzn-0	HTTP endpoint providing a REST interface to HCatalog.
hive-client	2.3.5-amzn-0	Hive command line client.
hive-hbase	2.3.5-amzn-0	Hive-hbase client.
hive-metastore-server	2.3.5-amzn-0	Service for accessing the Hive metastore, a semantic repository storing metadata for SQL on Hadoop operations.
hive-server2	2.3.5-amzn-0	Service for accepting Hive queries as web requests.
hue-server	4.4.0	Web application for analyzing data using Hadoop ecosystem applications
jupyterhub	0.9.6	Multi-user server for Jupyter notebooks
livy-server	0.6.0-incubating	REST interface for interacting with Apache Spark
nginx	1.12.1	nginx [engine x] is an HTTP and reverse proxy server
mahout-client	0.13.0	Library for machine learning.
mxnet	1.4.0	A flexible, scalable, and efficient library for deep learning.
mysql-server	5.5.54+	MySQL database server.
nvidia-cuda	9.2.88	Nvidia drivers and Cuda toolkit
oozie-client	5.1.0	Oozie command-line client.
oozie-server	5.1.0	Service for accepting Oozie workflow requests.
opencv	3.4.0	Open Source Computer Vision Library.
phoenix-library	4.14.1-HBase-1.4	The phoenix libraries for server and client
phoenix-query-server	4.14.1-HBase-1.4	A light weight server providing JDBC access as well as Protocol Buffers and JSON format access to the Avatica API
presto-coordinator	0.220	Service for accepting queries and managing query execution among presto-workers.
presto-worker	0.220	Service for executing pieces of a query.
pig-client	0.17.0	Pig command-line client.
r	3.4.1	The R Project for Statistical Computing
spark-client	2.4.3	Spark command-line clients.
spark-history-server	2.4.3	Web UI for viewing logged events for the lifetime of a completed Spark application.
spark-on-yarn	2.4.3	In-memory execution engine for YARN.
spark-yarn-slave	2.4.3	Apache Spark libraries needed by YARN slaves.
sqoop-client	1.4.7	Apache Sqoop command-line client.
tensorflow	1.13.1	TensorFlow open source software library for high performance numerical computation.
tez-on-yarn	0.9.2	The tez YARN application and libraries.
webserver	2.4.25+	Apache HTTP server.
zeppelin-server	0.8.1	Web-based notebook that enables interactive data analytics.
zookeeper-server	3.4.14	Centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
zookeeper-client	3.4.14	ZooKeeper command line client.

5.25.0 Configuration Classifications

emr-5.25.0 Classifications

Classifications	Description
capacity-scheduler	Change values in Hadoop's capacity-scheduler.xml file.
container-log4j	Change values in Hadoop YARN's container-log4j.properties file.
core-site	Change values in Hadoop's core-site.xml file.
emrfs-site	Change EMRFS settings.
flink-conf	Change flink-conf.yaml settings.
flink-log4j	Change Flink log4j.properties settings.
flink-log4j-yarn-session	Change Flink log4j-yarn-session.properties settings.
flink-log4j-cli	Change Flink log4j-cli.properties settings.
hadoop-env	Change values in the Hadoop environment for all Hadoop components.
hadoop-log4j	Change values in Hadoop's log4j.properties file.
hadoop-ssl-server	Change hadoop ssl server configuration
hadoop-ssl-client	Change hadoop ssl client configuration
hbase	Amazon EMR-curated settings for Apache HBase.
hbase-env	Change values in HBase's environment.
hbase-log4j	Change values in HBase's hbase-log4j.properties file.
hbase-metrics	Change values in HBase's hadoop-metrics2-hbase.properties file.
hbase-policy	Change values in HBase's hbase-policy.xml file.
hbase-site	Change values in HBase's hbase-site.xml file.
hdfs-encryption-zones	Configure HDFS encryption zones.
hdfs-site	Change values in HDFS's hdfs-site.xml.
hcatalog-env	Change values in HCatalog's environment.
hcatalog-server-jndi	Change values in HCatalog's jndi.properties.
hcatalog-server-proto-hive-site	Change values in HCatalog's proto-hive-site.xml.
hcatalog-webhcat-env	Change values in HCatalog WebHCat's environment.
hcatalog-webhcat-log4j2	Change values in HCatalog WebHCat's log4j2.properties.
hcatalog-webhcat-site	Change values in HCatalog WebHCat's webhcat-site.xml file.
hive-beeline-log4j2	Change values in Hive's beeline-log4j2.properties file.
hive-parquet-logging	Change values in Hive's parquet-logging.properties file.
hive-env	Change values in the Hive environment.
hive-exec-log4j2	Change values in Hive's hive-exec-log4j2.properties file.
hive-llap-daemon-log4j2	Change values in Hive's llap-daemon-log4j2.properties file.
hive-log4j2	Change values in Hive's hive-log4j2.properties file.
hive-site	Change values in Hive's hive-site.xml file
hiveserver2-site	Change values in Hive Server2's hiveserver2-site.xml file
hue-ini	Change values in Hue's ini file
httpfs-env	Change values in the HTTPFS environment.
httpfs-site	Change values in Hadoop's httpfs-site.xml file.
hadoop-kms-acls	Change values in Hadoop's kms-acls.xml file.
hadoop-kms-env	Change values in the Hadoop KMS environment.
hadoop-kms-log4j	Change values in Hadoop's kms-log4j.properties file.
hadoop-kms-site	Change values in Hadoop's kms-site.xml file.
jupyter-notebook-conf	Change values in Jupyter Notebook's jupyter_notebook_config.py file.
jupyter-hub-conf	Change values in JupyterHubs's jupyterhub_config.py file.
jupyter-s3-conf	Configure Jupyter Notebook S3 persistence.
jupyter-sparkmagic-conf	Change values in Sparkmagic's config.json file.
livy-conf	Change values in Livy's livy.conf file.
livy-env	Change values in the Livy environment.
livy-log4j	Change Livy log4j.properties settings.
mapred-env	Change values in the MapReduce application's environment.
mapred-site	Change values in the MapReduce application's mapred-site.xml file.
oozie-env	Change values in Oozie's environment.
oozie-log4j	Change values in Oozie's oozie-log4j.properties file.
oozie-site	Change values in Oozie's oozie-site.xml file.
phoenix-hbase-metrics	Change values in Phoenix's hadoop-metrics2-hbase.properties file.
phoenix-hbase-site	Change values in Phoenix's hbase-site.xml file.
phoenix-log4j	Change values in Phoenix's log4j.properties file.
phoenix-metrics	Change values in Phoenix's hadoop-metrics2-phoenix.properties file.
pig-env	Change values in the Pig environment.
pig-properties	Change values in Pig's pig.properties file.
pig-log4j	Change values in Pig's log4j.properties file.
presto-log	Change values in Presto's log.properties file.
presto-config	Change values in Presto's config.properties file.
presto-password-authenticator	Change values in Presto's password-authenticator.properties file.
presto-env	Change values in Presto's presto-env.sh file.
presto-node	Change values in Presto's node.properties file.
presto-connector-blackhole	Change values in Presto's blackhole.properties file.
presto-connector-cassandra	Change values in Presto's cassandra.properties file.
presto-connector-hive	Change values in Presto's hive.properties file.
presto-connector-jmx	Change values in Presto's jmx.properties file.
presto-connector-kafka	Change values in Presto's kafka.properties file.
presto-connector-localfile	Change values in Presto's localfile.properties file.
presto-connector-memory	Change values in Presto's memory.properties file.
presto-connector-mongodb	Change values in Presto's mongodb.properties file.
presto-connector-mysql	Change values in Presto's mysql.properties file.
presto-connector-postgresql	Change values in Presto's postgresql.properties file.
presto-connector-raptor	Change values in Presto's raptor.properties file.
presto-connector-redis	Change values in Presto's redis.properties file.
presto-connector-redshift	Change values in Presto's redshift.properties file.
presto-connector-tpch	Change values in Presto's tpch.properties file.
presto-connector-tpcds	Change values in Presto's tpcds.properties file.
recordserver-env	Change values in the EMR RecordServer environment.
recordserver-conf	Change values in EMR RecordServer's erver.properties file.
recordserver-log4j	Change values in EMR RecordServer's log4j.properties file.
spark	Amazon EMR-curated settings for Apache Spark.
spark-defaults	Change values in Spark's spark-defaults.conf file.
spark-env	Change values in the Spark environment.
spark-hive-site	Change values in Spark's hive-site.xml file
spark-log4j	Change values in Spark's log4j.properties file.
spark-metrics	Change values in Spark's metrics.properties file.
sqoop-env	Change values in Sqoop's environment.
sqoop-oraoop-site	Change values in Sqoop OraOop's oraoop-site.xml file.
sqoop-site	Change values in Sqoop's sqoop-site.xml file.
tez-site	Change values in Tez's tez-site.xml file.
yarn-env	Change values in the YARN environment.
yarn-site	Change values in YARN's yarn-site.xml file.
zeppelin-env	Change values in the Zeppelin environment.
zookeeper-config	Change values in ZooKeeper's zoo.cfg file.
zookeeper-log4j	Change values in ZooKeeper's log4j.properties file.

5.24.x

There are multiple releases within the 5.24 series. Choose a link below to see information for a specific release within this tab.

5.24.1 (Latest) | 5.24.0

Amazon EMR Release 5.24.1

Release 5.24.1 Application Versions

The diagram below depicts the application versions available in this release of Amazon EMR and the application versions in the preceding four Amazon EMR releases.

For a comprehensive history of application versions for each release of Amazon EMR, see the following diagrams:

Release 5.24.1 Release Notes

The following release notes include information for Amazon EMR release version 5.24.1. Changes are relative to 5.24.0.

Initial release date: June 26, 2019

Changes, Enhancements, and Resolved Issues

Updated the default Amazon Linux AMI for EMR to include important Linux kernel security updates, including the TCP SACK Denial of Service Issue (AWS-2019-005).

Release 5.24.1 Component Versions

Component	Version	Description
aws-sagemaker-spark-sdk	1.2.1	Amazon SageMaker Spark SDK
emr-ddb	4.9.0	Amazon DynamoDB connector for Hadoop ecosystem applications.
emr-goodies	2.8.0	Extra convenience libraries for the Hadoop ecosystem.
emr-kinesis	3.4.0	Amazon Kinesis connector for Hadoop ecosystem applications.
emr-s3-dist-cp	2.11.0	Distributed copy application optimized for Amazon S3.
emr-s3-select	1.3.0	EMR S3Select Connector
emrfs	2.33.0	Amazon S3 connector for Hadoop ecosystem applications.
flink-client	1.8.0	Apache Flink command line client scripts and applications.
ganglia-monitor	3.7.2	Embedded Ganglia agent for Hadoop ecosystem applications along with the Ganglia monitoring agent.
ganglia-metadata-collector	3.7.2	Ganglia metadata collector for aggregating metrics from Ganglia monitoring agents.
ganglia-web	3.7.1	Web application for viewing metrics collected by the Ganglia metadata collector.
hadoop-client	2.8.5-amzn-4	Hadoop command-line clients such as 'hdfs', 'hadoop', or 'yarn'.
hadoop-hdfs-datanode	2.8.5-amzn-4	HDFS node-level service for storing blocks.
hadoop-hdfs-library	2.8.5-amzn-4	HDFS command-line client and library
hadoop-hdfs-namenode	2.8.5-amzn-4	HDFS service for tracking file names and block locations.
hadoop-hdfs-journalnode	2.8.5-amzn-4	HDFS service for managing the Hadoop filesystem journal on HA clusters.
hadoop-httpfs-server	2.8.5-amzn-4	HTTP endpoint for HDFS operations.
hadoop-kms-server	2.8.5-amzn-4	Cryptographic key management server based on Hadoop's KeyProvider API.
hadoop-mapred	2.8.5-amzn-4	MapReduce execution engine libraries for running a MapReduce application.
hadoop-yarn-nodemanager	2.8.5-amzn-4	YARN service for managing containers on an individual node.
hadoop-yarn-resourcemanager	2.8.5-amzn-4	YARN service for allocating and managing cluster resources and distributed applications.
hadoop-yarn-timeline-server	2.8.5-amzn-4	Service for retrieving current and historical information for YARN applications.
hbase-hmaster	1.4.9	Service for an HBase cluster responsible for coordination of Regions and execution of administrative commands.
hbase-region-server	1.4.9	Service for serving one or more HBase regions.
hbase-client	1.4.9	HBase command-line client.
hbase-rest-server	1.4.9	Service providing a RESTful HTTP endpoint for HBase.
hbase-thrift-server	1.4.9	Service providing a Thrift endpoint to HBase.
hcatalog-client	2.3.4-amzn-2	The 'hcat' command line client for manipulating hcatalog-server.
hcatalog-server	2.3.4-amzn-2	Service providing HCatalog, a table and storage management layer for distributed applications.
hcatalog-webhcat-server	2.3.4-amzn-2	HTTP endpoint providing a REST interface to HCatalog.
hive-client	2.3.4-amzn-2	Hive command line client.
hive-hbase	2.3.4-amzn-2	Hive-hbase client.
hive-metastore-server	2.3.4-amzn-2	Service for accessing the Hive metastore, a semantic repository storing metadata for SQL on Hadoop operations.
hive-server2	2.3.4-amzn-2	Service for accepting Hive queries as web requests.
hue-server	4.4.0	Web application for analyzing data using Hadoop ecosystem applications
jupyterhub	0.9.6	Multi-user server for Jupyter notebooks
livy-server	0.6.0-incubating	REST interface for interacting with Apache Spark
nginx	1.12.1	nginx [engine x] is an HTTP and reverse proxy server
mahout-client	0.13.0	Library for machine learning.
mxnet	1.4.0	A flexible, scalable, and efficient library for deep learning.
mysql-server	5.5.54+	MySQL database server.
nvidia-cuda	9.2.88	Nvidia drivers and Cuda toolkit
oozie-client	5.1.0	Oozie command-line client.
oozie-server	5.1.0	Service for accepting Oozie workflow requests.
opencv	3.4.0	Open Source Computer Vision Library.
phoenix-library	4.14.1-HBase-1.4	The phoenix libraries for server and client
phoenix-query-server	4.14.1-HBase-1.4	A light weight server providing JDBC access as well as Protocol Buffers and JSON format access to the Avatica API
presto-coordinator	0.219	Service for accepting queries and managing query execution among presto-workers.
presto-worker	0.219	Service for executing pieces of a query.
pig-client	0.17.0	Pig command-line client.
r	3.4.1	The R Project for Statistical Computing
spark-client	2.4.2	Spark command-line clients.
spark-history-server	2.4.2	Web UI for viewing logged events for the lifetime of a completed Spark application.
spark-on-yarn	2.4.2	In-memory execution engine for YARN.
spark-yarn-slave	2.4.2	Apache Spark libraries needed by YARN slaves.
sqoop-client	1.4.7	Apache Sqoop command-line client.
tensorflow	1.12.0	TensorFlow open source software library for high performance numerical computation.
tez-on-yarn	0.9.1	The tez YARN application and libraries.
webserver	2.4.25+	Apache HTTP server.
zeppelin-server	0.8.1	Web-based notebook that enables interactive data analytics.
zookeeper-server	3.4.13	Centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
zookeeper-client	3.4.13	ZooKeeper command line client.

Release 5.24.1 Configuration Classifications

emr-5.24.1 Classifications

Classifications	Description
capacity-scheduler	Change values in Hadoop's capacity-scheduler.xml file.
container-log4j	Change values in Hadoop YARN's container-log4j.properties file.
core-site	Change values in Hadoop's core-site.xml file.
emrfs-site	Change EMRFS settings.
flink-conf	Change flink-conf.yaml settings.
flink-log4j	Change Flink log4j.properties settings.
flink-log4j-yarn-session	Change Flink log4j-yarn-session.properties settings.
flink-log4j-cli	Change Flink log4j-cli.properties settings.
hadoop-env	Change values in the Hadoop environment for all Hadoop components.
hadoop-log4j	Change values in Hadoop's log4j.properties file.
hadoop-ssl-server	Change hadoop ssl server configuration
hadoop-ssl-client	Change hadoop ssl client configuration
hbase	Amazon EMR-curated settings for Apache HBase.
hbase-env	Change values in HBase's environment.
hbase-log4j	Change values in HBase's hbase-log4j.properties file.
hbase-metrics	Change values in HBase's hadoop-metrics2-hbase.properties file.
hbase-policy	Change values in HBase's hbase-policy.xml file.
hbase-site	Change values in HBase's hbase-site.xml file.
hdfs-encryption-zones	Configure HDFS encryption zones.
hdfs-site	Change values in HDFS's hdfs-site.xml.
hcatalog-env	Change values in HCatalog's environment.
hcatalog-server-jndi	Change values in HCatalog's jndi.properties.
hcatalog-server-proto-hive-site	Change values in HCatalog's proto-hive-site.xml.
hcatalog-webhcat-env	Change values in HCatalog WebHCat's environment.
hcatalog-webhcat-log4j2	Change values in HCatalog WebHCat's log4j2.properties.
hcatalog-webhcat-site	Change values in HCatalog WebHCat's webhcat-site.xml file.
hive-beeline-log4j2	Change values in Hive's beeline-log4j2.properties file.
hive-parquet-logging	Change values in Hive's parquet-logging.properties file.
hive-env	Change values in the Hive environment.
hive-exec-log4j2	Change values in Hive's hive-exec-log4j2.properties file.
hive-llap-daemon-log4j2	Change values in Hive's llap-daemon-log4j2.properties file.
hive-log4j2	Change values in Hive's hive-log4j2.properties file.
hive-site	Change values in Hive's hive-site.xml file
hiveserver2-site	Change values in Hive Server2's hiveserver2-site.xml file
hue-ini	Change values in Hue's ini file
httpfs-env	Change values in the HTTPFS environment.
httpfs-site	Change values in Hadoop's httpfs-site.xml file.
hadoop-kms-acls	Change values in Hadoop's kms-acls.xml file.
hadoop-kms-env	Change values in the Hadoop KMS environment.
hadoop-kms-log4j	Change values in Hadoop's kms-log4j.properties file.
hadoop-kms-site	Change values in Hadoop's kms-site.xml file.
jupyter-notebook-conf	Change values in Jupyter Notebook's jupyter_notebook_config.py file.
jupyter-hub-conf	Change values in JupyterHubs's jupyterhub_config.py file.
jupyter-s3-conf	Configure Jupyter Notebook S3 persistence.
jupyter-sparkmagic-conf	Change values in Sparkmagic's config.json file.
livy-conf	Change values in Livy's livy.conf file.
livy-env	Change values in the Livy environment.
livy-log4j	Change Livy log4j.properties settings.
mapred-env	Change values in the MapReduce application's environment.
mapred-site	Change values in the MapReduce application's mapred-site.xml file.
oozie-env	Change values in Oozie's environment.
oozie-log4j	Change values in Oozie's oozie-log4j.properties file.
oozie-site	Change values in Oozie's oozie-site.xml file.
phoenix-hbase-metrics	Change values in Phoenix's hadoop-metrics2-hbase.properties file.
phoenix-hbase-site	Change values in Phoenix's hbase-site.xml file.
phoenix-log4j	Change values in Phoenix's log4j.properties file.
phoenix-metrics	Change values in Phoenix's hadoop-metrics2-phoenix.properties file.
pig-env	Change values in the Pig environment.
pig-properties	Change values in Pig's pig.properties file.
pig-log4j	Change values in Pig's log4j.properties file.
presto-log	Change values in Presto's log.properties file.
presto-config	Change values in Presto's config.properties file.
presto-password-authenticator	Change values in Presto's password-authenticator.properties file.
presto-env	Change values in Presto's presto-env.sh file.
presto-node	Change values in Presto's node.properties file.
presto-connector-blackhole	Change values in Presto's blackhole.properties file.
presto-connector-cassandra	Change values in Presto's cassandra.properties file.
presto-connector-hive	Change values in Presto's hive.properties file.
presto-connector-jmx	Change values in Presto's jmx.properties file.
presto-connector-kafka	Change values in Presto's kafka.properties file.
presto-connector-localfile	Change values in Presto's localfile.properties file.
presto-connector-memory	Change values in Presto's memory.properties file.
presto-connector-mongodb	Change values in Presto's mongodb.properties file.
presto-connector-mysql	Change values in Presto's mysql.properties file.
presto-connector-postgresql	Change values in Presto's postgresql.properties file.
presto-connector-raptor	Change values in Presto's raptor.properties file.
presto-connector-redis	Change values in Presto's redis.properties file.
presto-connector-redshift	Change values in Presto's redshift.properties file.
presto-connector-tpch	Change values in Presto's tpch.properties file.
presto-connector-tpcds	Change values in Presto's tpcds.properties file.
spark	Amazon EMR-curated settings for Apache Spark.
spark-defaults	Change values in Spark's spark-defaults.conf file.
spark-env	Change values in the Spark environment.
spark-hive-site	Change values in Spark's hive-site.xml file
spark-log4j	Change values in Spark's log4j.properties file.
spark-metrics	Change values in Spark's metrics.properties file.
sqoop-env	Change values in Sqoop's environment.
sqoop-oraoop-site	Change values in Sqoop OraOop's oraoop-site.xml file.
sqoop-site	Change values in Sqoop's sqoop-site.xml file.
tez-site	Change values in Tez's tez-site.xml file.
yarn-env	Change values in the YARN environment.
yarn-site	Change values in YARN's yarn-site.xml file.
zeppelin-env	Change values in the Zeppelin environment.
zookeeper-config	Change values in ZooKeeper's zoo.cfg file.
zookeeper-log4j	Change values in ZooKeeper's log4j.properties file.

Amazon EMR Release 5.24.0

Release 5.24.0 Application Versions

The diagram below depicts the application versions available in this release of Amazon EMR and the application versions in the preceding four Amazon EMR releases.

For a comprehensive history of application versions for each release of Amazon EMR, see the following diagrams:

Release 5.24.0 Release Notes

The following release notes include information for Amazon EMR release version 5.24.0. Changes are relative to 5.23.0.

Initial release date: June 11, 2019

Last updated date: August 5, 2019

Upgrades

Flink 1.8.0
Hue 4.4.0
JupyterHub 0.9.6
Livy 0.6.0
MxNet 1.4.0
Presto 0.219
Spark 2.4.2
AWS SDK for Java 1.11.546
Connectors and drivers:
- DynamoDB Connector 4.9.0
- MariaDB Connector 2.4.1
- Amazon Redshift JDBC Driver 1.2.27.1051

Changes, Enhancements, and Resolved Issues

Spark
- Added optimization to dynamically prune partitions. The optimization is disabled by default. To enable it, set the Spark configuration parameter spark.sql.dynamicPartitionPruning.enabled to true.
- Improved performance of INTERSECT queries. This optimization is disabled by default. To enable it, set the Spark configuration parameter spark.sql.optimizer.distinctBeforeIntersect.enabled to true.
- Added optimization to flatten scalar subqueries with aggregates that use the same relation. The optimization is disabled by default. To enable it, set the Spark configuration parameter spark.sql.optimizer.flattenScalarSubqueriesWithAggregates.enabled to true.
- Improved whole stage code generation.
For more information, see Optimizing Spark Performance.
Local Disk Encryption and S3 Encryption Improvements in Security Configurations (August 5, 2019)
- Separated Amazon S3 encryption settings from local disk encryption settings in security configuration setup.
- Added an option to enable EBS encryption. Selecting this option encrypts the root device volume in addition to storage volumes. Previous versions required using a custom AMI to encrypt the root device volume.
- For more information, see Encryption Options in the Amazon EMR Management Guide.

Release 5.24.0 Component Versions

Component	Version	Description
aws-sagemaker-spark-sdk	1.2.1	Amazon SageMaker Spark SDK
emr-ddb	4.9.0	Amazon DynamoDB connector for Hadoop ecosystem applications.
emr-goodies	2.8.0	Extra convenience libraries for the Hadoop ecosystem.
emr-kinesis	3.4.0	Amazon Kinesis connector for Hadoop ecosystem applications.
emr-s3-dist-cp	2.11.0	Distributed copy application optimized for Amazon S3.
emr-s3-select	1.3.0	EMR S3Select Connector
emrfs	2.33.0	Amazon S3 connector for Hadoop ecosystem applications.
flink-client	1.8.0	Apache Flink command line client scripts and applications.
ganglia-monitor	3.7.2	Embedded Ganglia agent for Hadoop ecosystem applications along with the Ganglia monitoring agent.
ganglia-metadata-collector	3.7.2	Ganglia metadata collector for aggregating metrics from Ganglia monitoring agents.
ganglia-web	3.7.1	Web application for viewing metrics collected by the Ganglia metadata collector.
hadoop-client	2.8.5-amzn-4	Hadoop command-line clients such as 'hdfs', 'hadoop', or 'yarn'.
hadoop-hdfs-datanode	2.8.5-amzn-4	HDFS node-level service for storing blocks.
hadoop-hdfs-library	2.8.5-amzn-4	HDFS command-line client and library
hadoop-hdfs-namenode	2.8.5-amzn-4	HDFS service for tracking file names and block locations.
hadoop-hdfs-journalnode	2.8.5-amzn-4	HDFS service for managing the Hadoop filesystem journal on HA clusters.
hadoop-httpfs-server	2.8.5-amzn-4	HTTP endpoint for HDFS operations.
hadoop-kms-server	2.8.5-amzn-4	Cryptographic key management server based on Hadoop's KeyProvider API.
hadoop-mapred	2.8.5-amzn-4	MapReduce execution engine libraries for running a MapReduce application.
hadoop-yarn-nodemanager	2.8.5-amzn-4	YARN service for managing containers on an individual node.
hadoop-yarn-resourcemanager	2.8.5-amzn-4	YARN service for allocating and managing cluster resources and distributed applications.
hadoop-yarn-timeline-server	2.8.5-amzn-4	Service for retrieving current and historical information for YARN applications.
hbase-hmaster	1.4.9	Service for an HBase cluster responsible for coordination of Regions and execution of administrative commands.
hbase-region-server	1.4.9	Service for serving one or more HBase regions.
hbase-client	1.4.9	HBase command-line client.
hbase-rest-server	1.4.9	Service providing a RESTful HTTP endpoint for HBase.
hbase-thrift-server	1.4.9	Service providing a Thrift endpoint to HBase.
hcatalog-client	2.3.4-amzn-2	The 'hcat' command line client for manipulating hcatalog-server.
hcatalog-server	2.3.4-amzn-2	Service providing HCatalog, a table and storage management layer for distributed applications.
hcatalog-webhcat-server	2.3.4-amzn-2	HTTP endpoint providing a REST interface to HCatalog.
hive-client	2.3.4-amzn-2	Hive command line client.
hive-hbase	2.3.4-amzn-2	Hive-hbase client.
hive-metastore-server	2.3.4-amzn-2	Service for accessing the Hive metastore, a semantic repository storing metadata for SQL on Hadoop operations.
hive-server2	2.3.4-amzn-2	Service for accepting Hive queries as web requests.
hue-server	4.4.0	Web application for analyzing data using Hadoop ecosystem applications
jupyterhub	0.9.6	Multi-user server for Jupyter notebooks
livy-server	0.6.0-incubating	REST interface for interacting with Apache Spark
nginx	1.12.1	nginx [engine x] is an HTTP and reverse proxy server
mahout-client	0.13.0	Library for machine learning.
mxnet	1.4.0	A flexible, scalable, and efficient library for deep learning.
mysql-server	5.5.54+	MySQL database server.
nvidia-cuda	9.2.88	Nvidia drivers and Cuda toolkit
oozie-client	5.1.0	Oozie command-line client.
oozie-server	5.1.0	Service for accepting Oozie workflow requests.
opencv	3.4.0	Open Source Computer Vision Library.
phoenix-library	4.14.1-HBase-1.4	The phoenix libraries for server and client
phoenix-query-server	4.14.1-HBase-1.4	A light weight server providing JDBC access as well as Protocol Buffers and JSON format access to the Avatica API
presto-coordinator	0.219	Service for accepting queries and managing query execution among presto-workers.
presto-worker	0.219	Service for executing pieces of a query.
pig-client	0.17.0	Pig command-line client.
r	3.4.1	The R Project for Statistical Computing
spark-client	2.4.2	Spark command-line clients.
spark-history-server	2.4.2	Web UI for viewing logged events for the lifetime of a completed Spark application.
spark-on-yarn	2.4.2	In-memory execution engine for YARN.
spark-yarn-slave	2.4.2	Apache Spark libraries needed by YARN slaves.
sqoop-client	1.4.7	Apache Sqoop command-line client.
tensorflow	1.12.0	TensorFlow open source software library for high performance numerical computation.
tez-on-yarn	0.9.1	The tez YARN application and libraries.
webserver	2.4.25+	Apache HTTP server.
zeppelin-server	0.8.1	Web-based notebook that enables interactive data analytics.
zookeeper-server	3.4.13	Centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
zookeeper-client	3.4.13	ZooKeeper command line client.

Release 5.24.0 Configuration Classifications

emr-5.24.0 Classifications

Classifications	Description
capacity-scheduler	Change values in Hadoop's capacity-scheduler.xml file.
container-log4j	Change values in Hadoop YARN's container-log4j.properties file.
core-site	Change values in Hadoop's core-site.xml file.
emrfs-site	Change EMRFS settings.
flink-conf	Change flink-conf.yaml settings.
flink-log4j	Change Flink log4j.properties settings.
flink-log4j-yarn-session	Change Flink log4j-yarn-session.properties settings.
flink-log4j-cli	Change Flink log4j-cli.properties settings.
hadoop-env	Change values in the Hadoop environment for all Hadoop components.
hadoop-log4j	Change values in Hadoop's log4j.properties file.
hadoop-ssl-server	Change hadoop ssl server configuration
hadoop-ssl-client	Change hadoop ssl client configuration
hbase	Amazon EMR-curated settings for Apache HBase.
hbase-env	Change values in HBase's environment.
hbase-log4j	Change values in HBase's hbase-log4j.properties file.
hbase-metrics	Change values in HBase's hadoop-metrics2-hbase.properties file.
hbase-policy	Change values in HBase's hbase-policy.xml file.
hbase-site	Change values in HBase's hbase-site.xml file.
hdfs-encryption-zones	Configure HDFS encryption zones.
hdfs-site	Change values in HDFS's hdfs-site.xml.
hcatalog-env	Change values in HCatalog's environment.
hcatalog-server-jndi	Change values in HCatalog's jndi.properties.
hcatalog-server-proto-hive-site	Change values in HCatalog's proto-hive-site.xml.
hcatalog-webhcat-env	Change values in HCatalog WebHCat's environment.
hcatalog-webhcat-log4j2	Change values in HCatalog WebHCat's log4j2.properties.
hcatalog-webhcat-site	Change values in HCatalog WebHCat's webhcat-site.xml file.
hive-beeline-log4j2	Change values in Hive's beeline-log4j2.properties file.
hive-parquet-logging	Change values in Hive's parquet-logging.properties file.
hive-env	Change values in the Hive environment.
hive-exec-log4j2	Change values in Hive's hive-exec-log4j2.properties file.
hive-llap-daemon-log4j2	Change values in Hive's llap-daemon-log4j2.properties file.
hive-log4j2	Change values in Hive's hive-log4j2.properties file.
hive-site	Change values in Hive's hive-site.xml file
hiveserver2-site	Change values in Hive Server2's hiveserver2-site.xml file
hue-ini	Change values in Hue's ini file
httpfs-env	Change values in the HTTPFS environment.
httpfs-site	Change values in Hadoop's httpfs-site.xml file.
hadoop-kms-acls	Change values in Hadoop's kms-acls.xml file.
hadoop-kms-env	Change values in the Hadoop KMS environment.
hadoop-kms-log4j	Change values in Hadoop's kms-log4j.properties file.
hadoop-kms-site	Change values in Hadoop's kms-site.xml file.
jupyter-notebook-conf	Change values in Jupyter Notebook's jupyter_notebook_config.py file.
jupyter-hub-conf	Change values in JupyterHubs's jupyterhub_config.py file.
jupyter-s3-conf	Configure Jupyter Notebook S3 persistence.
jupyter-sparkmagic-conf	Change values in Sparkmagic's config.json file.
livy-conf	Change values in Livy's livy.conf file.
livy-env	Change values in the Livy environment.
livy-log4j	Change Livy log4j.properties settings.
mapred-env	Change values in the MapReduce application's environment.
mapred-site	Change values in the MapReduce application's mapred-site.xml file.
oozie-env	Change values in Oozie's environment.
oozie-log4j	Change values in Oozie's oozie-log4j.properties file.
oozie-site	Change values in Oozie's oozie-site.xml file.
phoenix-hbase-metrics	Change values in Phoenix's hadoop-metrics2-hbase.properties file.
phoenix-hbase-site	Change values in Phoenix's hbase-site.xml file.
phoenix-log4j	Change values in Phoenix's log4j.properties file.
phoenix-metrics	Change values in Phoenix's hadoop-metrics2-phoenix.properties file.
pig-env	Change values in the Pig environment.
pig-properties	Change values in Pig's pig.properties file.
pig-log4j	Change values in Pig's log4j.properties file.
presto-log	Change values in Presto's log.properties file.
presto-config	Change values in Presto's config.properties file.
presto-password-authenticator	Change values in Presto's password-authenticator.properties file.
presto-env	Change values in Presto's presto-env.sh file.
presto-node	Change values in Presto's node.properties file.
presto-connector-blackhole	Change values in Presto's blackhole.properties file.
presto-connector-cassandra	Change values in Presto's cassandra.properties file.
presto-connector-hive	Change values in Presto's hive.properties file.
presto-connector-jmx	Change values in Presto's jmx.properties file.
presto-connector-kafka	Change values in Presto's kafka.properties file.
presto-connector-localfile	Change values in Presto's localfile.properties file.
presto-connector-memory	Change values in Presto's memory.properties file.
presto-connector-mongodb	Change values in Presto's mongodb.properties file.
presto-connector-mysql	Change values in Presto's mysql.properties file.
presto-connector-postgresql	Change values in Presto's postgresql.properties file.
presto-connector-raptor	Change values in Presto's raptor.properties file.
presto-connector-redis	Change values in Presto's redis.properties file.
presto-connector-redshift	Change values in Presto's redshift.properties file.
presto-connector-tpch	Change values in Presto's tpch.properties file.
presto-connector-tpcds	Change values in Presto's tpcds.properties file.
spark	Amazon EMR-curated settings for Apache Spark.
spark-defaults	Change values in Spark's spark-defaults.conf file.
spark-env	Change values in the Spark environment.
spark-hive-site	Change values in Spark's hive-site.xml file
spark-log4j	Change values in Spark's log4j.properties file.
spark-metrics	Change values in Spark's metrics.properties file.
sqoop-env	Change values in Sqoop's environment.
sqoop-oraoop-site	Change values in Sqoop OraOop's oraoop-site.xml file.
sqoop-site	Change values in Sqoop's sqoop-site.xml file.
tez-site	Change values in Tez's tez-site.xml file.
yarn-env	Change values in the YARN environment.
yarn-site	Change values in YARN's yarn-site.xml file.
zeppelin-env	Change values in the Zeppelin environment.
zookeeper-config	Change values in ZooKeeper's zoo.cfg file.
zookeeper-log4j	Change values in ZooKeeper's log4j.properties file.

5.23.0

Amazon EMR Release 5.23.0

5.23.0 Application Versions

The diagram below depicts the application versions available in this release of Amazon EMR and the application versions in the preceding four Amazon EMR releases.

For a comprehensive history of application versions for each release of Amazon EMR, see the following diagrams:

5.23.0 Release Notes

The following release notes include information for Amazon EMR release version 5.23.0. Changes are relative to 5.22.0.

Initial release date: April 01, 2019

Last updated date: April 30, 2019

Upgrades

AWS SDK for Java 1.11.519

New Features

(April 30, 2019) With Amazon EMR 5.23.0 and later, you can launch a cluster with three master nodes to support high availability of applications like YARN Resource Manager, HDFS Name Node, Spark, Hive, and Ganglia. The master node is no longer a potential single point of failure with this feature. If one of the master nodes fails, Amazon EMR automatically fails over to a standby master node and replaces the failed master node with a new one with the same configuration and bootstrap actions. For more information, see Plan and Configure Master Nodes.

Known Issues

Tez UI (Fixed in Amazon EMR release version 5.26.0)

Tez UI does not work on an EMR cluster with multiple master nodes.
Hue (Fixed in Amazon EMR release version 5.24.0)
- Hue running on Amazon EMR does not support Solr. Beginning with Amazon EMR release version 5.20.0, a misconfiguration issue causes Solr to be enabled and a harmless error message to appear similar to the following:
  
  Solr server could not be contacted properly: HTTPConnectionPool('host=ip-xx-xx-xx-xx.ec2.internal', port=1978): Max retries exceeded with url: /solr/admin/info/system?user.name=hue&doAs=administrator&wt=json (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',))
  
  To prevent the Solr error message from appearing:
  1. Connect to the master node command line using SSH.
  2. Use a text editor to open the hue.ini file. For example:
    
    sudo vim /etc/hue/conf/hue.ini
  3. Search for the term "appblacklist" and modify the line to the following:
    
    appblacklist = search
  4. Save your changes and restart Hue as shown in the following example:
    
    sudo stop hue; sudo start hue

5.23.0 Component Versions

Component	Version	Description
aws-sagemaker-spark-sdk	1.2.1	Amazon SageMaker Spark SDK
emr-ddb	4.8.0	Amazon DynamoDB connector for Hadoop ecosystem applications.
emr-goodies	2.7.0	Extra convenience libraries for the Hadoop ecosystem.
emr-kinesis	3.4.0	Amazon Kinesis connector for Hadoop ecosystem applications.
emr-s3-dist-cp	2.11.0	Distributed copy application optimized for Amazon S3.
emr-s3-select	1.2.0	EMR S3Select Connector
emrfs	2.32.0	Amazon S3 connector for Hadoop ecosystem applications.
flink-client	1.7.1	Apache Flink command line client scripts and applications.
ganglia-monitor	3.7.2	Embedded Ganglia agent for Hadoop ecosystem applications along with the Ganglia monitoring agent.
ganglia-metadata-collector	3.7.2	Ganglia metadata collector for aggregating metrics from Ganglia monitoring agents.
ganglia-web	3.7.1	Web application for viewing metrics collected by the Ganglia metadata collector.
hadoop-client	2.8.5-amzn-3	Hadoop command-line clients such as 'hdfs', 'hadoop', or 'yarn'.
hadoop-hdfs-datanode	2.8.5-amzn-3	HDFS node-level service for storing blocks.
hadoop-hdfs-library	2.8.5-amzn-3	HDFS command-line client and library
hadoop-hdfs-namenode	2.8.5-amzn-3	HDFS service for tracking file names and block locations.
hadoop-hdfs-journalnode	2.8.5-amzn-3	HDFS service for managing the Hadoop filesystem journal on HA clusters.
hadoop-httpfs-server	2.8.5-amzn-3	HTTP endpoint for HDFS operations.
hadoop-kms-server	2.8.5-amzn-3	Cryptographic key management server based on Hadoop's KeyProvider API.
hadoop-mapred	2.8.5-amzn-3	MapReduce execution engine libraries for running a MapReduce application.
hadoop-yarn-nodemanager	2.8.5-amzn-3	YARN service for managing containers on an individual node.
hadoop-yarn-resourcemanager	2.8.5-amzn-3	YARN service for allocating and managing cluster resources and distributed applications.
hadoop-yarn-timeline-server	2.8.5-amzn-3	Service for retrieving current and historical information for YARN applications.
hbase-hmaster	1.4.9	Service for an HBase cluster responsible for coordination of Regions and execution of administrative commands.
hbase-region-server	1.4.9	Service for serving one or more HBase regions.
hbase-client	1.4.9	HBase command-line client.
hbase-rest-server	1.4.9	Service providing a RESTful HTTP endpoint for HBase.
hbase-thrift-server	1.4.9	Service providing a Thrift endpoint to HBase.
hcatalog-client	2.3.4-amzn-1	The 'hcat' command line client for manipulating hcatalog-server.
hcatalog-server	2.3.4-amzn-1	Service providing HCatalog, a table and storage management layer for distributed applications.
hcatalog-webhcat-server	2.3.4-amzn-1	HTTP endpoint providing a REST interface to HCatalog.
hive-client	2.3.4-amzn-1	Hive command line client.
hive-hbase	2.3.4-amzn-1	Hive-hbase client.
hive-metastore-server	2.3.4-amzn-1	Service for accessing the Hive metastore, a semantic repository storing metadata for SQL on Hadoop operations.
hive-server2	2.3.4-amzn-1	Service for accepting Hive queries as web requests.
hue-server	4.3.0	Web application for analyzing data using Hadoop ecosystem applications
jupyterhub	0.9.4	Multi-user server for Jupyter notebooks
livy-server	0.5.0-incubating	REST interface for interacting with Apache Spark
nginx	1.12.1	nginx [engine x] is an HTTP and reverse proxy server
mahout-client	0.13.0	Library for machine learning.
mxnet	1.3.1	A flexible, scalable, and efficient library for deep learning.
mysql-server	5.5.54+	MySQL database server.
nvidia-cuda	9.2.88	Nvidia drivers and Cuda toolkit
oozie-client	5.1.0	Oozie command-line client.
oozie-server	5.1.0	Service for accepting Oozie workflow requests.
opencv	3.4.0	Open Source Computer Vision Library.
phoenix-library	4.14.1-HBase-1.4	The phoenix libraries for server and client
phoenix-query-server	4.14.1-HBase-1.4	A light weight server providing JDBC access as well as Protocol Buffers and JSON format access to the Avatica API
presto-coordinator	0.215	Service for accepting queries and managing query execution among presto-workers.
presto-worker	0.215	Service for executing pieces of a query.
pig-client	0.17.0	Pig command-line client.
r	3.4.1	The R Project for Statistical Computing
spark-client	2.4.0	Spark command-line clients.
spark-history-server	2.4.0	Web UI for viewing logged events for the lifetime of a completed Spark application.
spark-on-yarn	2.4.0	In-memory execution engine for YARN.
spark-yarn-slave	2.4.0	Apache Spark libraries needed by YARN slaves.
sqoop-client	1.4.7	Apache Sqoop command-line client.
tensorflow	1.12.0	TensorFlow open source software library for high performance numerical computation.
tez-on-yarn	0.9.1	The tez YARN application and libraries.
webserver	2.4.25+	Apache HTTP server.
zeppelin-server	0.8.1	Web-based notebook that enables interactive data analytics.
zookeeper-server	3.4.13	Centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
zookeeper-client	3.4.13	ZooKeeper command line client.

5.23.0 Configuration Classifications

emr-5.23.0 Classifications

Classifications	Description
capacity-scheduler	Change values in Hadoop's capacity-scheduler.xml file.
container-log4j	Change values in Hadoop YARN's container-log4j.properties file.
core-site	Change values in Hadoop's core-site.xml file.
emrfs-site	Change EMRFS settings.
flink-conf	Change flink-conf.yaml settings.
flink-log4j	Change Flink log4j.properties settings.
flink-log4j-yarn-session	Change Flink log4j-yarn-session.properties settings.
flink-log4j-cli	Change Flink log4j-cli.properties settings.
hadoop-env	Change values in the Hadoop environment for all Hadoop components.
hadoop-log4j	Change values in Hadoop's log4j.properties file.
hadoop-ssl-server	Change hadoop ssl server configuration
hadoop-ssl-client	Change hadoop ssl client configuration
hbase	Amazon EMR-curated settings for Apache HBase.
hbase-env	Change values in HBase's environment.
hbase-log4j	Change values in HBase's hbase-log4j.properties file.
hbase-metrics	Change values in HBase's hadoop-metrics2-hbase.properties file.
hbase-policy	Change values in HBase's hbase-policy.xml file.
hbase-site	Change values in HBase's hbase-site.xml file.
hdfs-encryption-zones	Configure HDFS encryption zones.
hdfs-site	Change values in HDFS's hdfs-site.xml.
hcatalog-env	Change values in HCatalog's environment.
hcatalog-server-jndi	Change values in HCatalog's jndi.properties.
hcatalog-server-proto-hive-site	Change values in HCatalog's proto-hive-site.xml.
hcatalog-webhcat-env	Change values in HCatalog WebHCat's environment.
hcatalog-webhcat-log4j2	Change values in HCatalog WebHCat's log4j2.properties.
hcatalog-webhcat-site	Change values in HCatalog WebHCat's webhcat-site.xml file.
hive-beeline-log4j2	Change values in Hive's beeline-log4j2.properties file.
hive-parquet-logging	Change values in Hive's parquet-logging.properties file.
hive-env	Change values in the Hive environment.
hive-exec-log4j2	Change values in Hive's hive-exec-log4j2.properties file.
hive-llap-daemon-log4j2	Change values in Hive's llap-daemon-log4j2.properties file.
hive-log4j2	Change values in Hive's hive-log4j2.properties file.
hive-site	Change values in Hive's hive-site.xml file
hiveserver2-site	Change values in Hive Server2's hiveserver2-site.xml file
hue-ini	Change values in Hue's ini file
httpfs-env	Change values in the HTTPFS environment.
httpfs-site	Change values in Hadoop's httpfs-site.xml file.
hadoop-kms-acls	Change values in Hadoop's kms-acls.xml file.
hadoop-kms-env	Change values in the Hadoop KMS environment.
hadoop-kms-log4j	Change values in Hadoop's kms-log4j.properties file.
hadoop-kms-site	Change values in Hadoop's kms-site.xml file.
jupyter-notebook-conf	Change values in Jupyter Notebook's jupyter_notebook_config.py file.
jupyter-hub-conf	Change values in JupyterHubs's jupyterhub_config.py file.
jupyter-s3-conf	Configure Jupyter Notebook S3 persistence.
jupyter-sparkmagic-conf	Change values in Sparkmagic's config.json file.
livy-conf	Change values in Livy's livy.conf file.
livy-env	Change values in the Livy environment.
livy-log4j	Change Livy log4j.properties settings.
mapred-env	Change values in the MapReduce application's environment.
mapred-site	Change values in the MapReduce application's mapred-site.xml file.
oozie-env	Change values in Oozie's environment.
oozie-log4j	Change values in Oozie's oozie-log4j.properties file.
oozie-site	Change values in Oozie's oozie-site.xml file.
phoenix-hbase-metrics	Change values in Phoenix's hadoop-metrics2-hbase.properties file.
phoenix-hbase-site	Change values in Phoenix's hbase-site.xml file.
phoenix-log4j	Change values in Phoenix's log4j.properties file.
phoenix-metrics	Change values in Phoenix's hadoop-metrics2-phoenix.properties file.
pig-env	Change values in the Pig environment.
pig-properties	Change values in Pig's pig.properties file.
pig-log4j	Change values in Pig's log4j.properties file.
presto-log	Change values in Presto's log.properties file.
presto-config	Change values in Presto's config.properties file.
presto-password-authenticator	Change values in Presto's password-authenticator.properties file.
presto-env	Change values in Presto's presto-env.sh file.
presto-node	Change values in Presto's node.properties file.
presto-connector-blackhole	Change values in Presto's blackhole.properties file.
presto-connector-cassandra	Change values in Presto's cassandra.properties file.
presto-connector-hive	Change values in Presto's hive.properties file.
presto-connector-jmx	Change values in Presto's jmx.properties file.
presto-connector-kafka	Change values in Presto's kafka.properties file.
presto-connector-localfile	Change values in Presto's localfile.properties file.
presto-connector-memory	Change values in Presto's memory.properties file.
presto-connector-mongodb	Change values in Presto's mongodb.properties file.
presto-connector-mysql	Change values in Presto's mysql.properties file.
presto-connector-postgresql	Change values in Presto's postgresql.properties file.
presto-connector-raptor	Change values in Presto's raptor.properties file.
presto-connector-redis	Change values in Presto's redis.properties file.
presto-connector-redshift	Change values in Presto's redshift.properties file.
presto-connector-tpch	Change values in Presto's tpch.properties file.
presto-connector-tpcds	Change values in Presto's tpcds.properties file.
spark	Amazon EMR-curated settings for Apache Spark.
spark-defaults	Change values in Spark's spark-defaults.conf file.
spark-env	Change values in the Spark environment.
spark-hive-site	Change values in Spark's hive-site.xml file
spark-log4j	Change values in Spark's log4j.properties file.
spark-metrics	Change values in Spark's metrics.properties file.
sqoop-env	Change values in Sqoop's environment.
sqoop-oraoop-site	Change values in Sqoop OraOop's oraoop-site.xml file.
sqoop-site	Change values in Sqoop's sqoop-site.xml file.
tez-site	Change values in Tez's tez-site.xml file.
yarn-env	Change values in the YARN environment.
yarn-site	Change values in YARN's yarn-site.xml file.
zeppelin-env	Change values in the Zeppelin environment.
zookeeper-config	Change values in ZooKeeper's zoo.cfg file.
zookeeper-log4j	Change values in ZooKeeper's log4j.properties file.

5.22.0

Amazon EMR Release 5.22.0

5.22.0 Application Versions

The diagram below depicts the application versions available in this release of Amazon EMR and the application versions in the preceding four Amazon EMR releases.

For a comprehensive history of application versions for each release of Amazon EMR, see the following diagrams:

5.22.0 Release Notes

The following release notes include information for Amazon EMR release version 5.22.0. Changes are relative to 5.21.0.

Important

Beginning with Amazon EMR release version 5.22.0, Amazon EMR uses AWS Signature Version 4 exclusively to authenticate requests to Amazon S3. Earlier Amazon EMR release versions use AWS Signature Version 2 in some cases, unless the release notes indicate that Signature Version 4 is used exclusively. For more information, see Authenticating Requests (AWS Signature Version 4) and Authenticating Requests (AWS Signature Version 2) in the Amazon Simple Storage Service Developer Guide.

Initial release date: March 20, 2019

Upgrades

Flink 1.7.1
HBase 1.4.9
Oozie 5.1.0
Phoenix 4.14.1
Zeppelin 0.8.1
Connectors and drivers:
- DynamoDB Connector 4.8.0
- MariaDB Connector 2.2.6
- Amazon Redshift JDBC Driver 1.2.20.1043

New Features

Modified the default EBS configuration for EC2 instance types with EBS-only storage. When you create a cluster using Amazon EMR release version 5.22.0 and later, the default amount of EBS storage increases based on the size of the instance. In addition, we split increased storage across multiple volumes, giving increased IOPS performance. If you want to use a different EBS instance storage configuration, you can specify it when you create an EMR cluster or add nodes to an existing cluster. For more information about the amount of storage and number of volumes allocated by default for each instance type, see Default EBS Storage for Instances in the Amazon EMR Management Guide.

Changes, Enhancements, and Resolved Issues

Spark
- Introduced a new configuration property for Spark on YARN, spark.yarn.executor.memoryOverheadFactor. The value of this property is a scale factor that sets the value of memory overhead to a percentage of executor memory, with a minimum of 384 MB. If memory overhead is set explicitly using spark.yarn.executor.memoryOverhead, this property has no effect. The default value is 0.1875, representing 18.75%. This default for Amazon EMR leaves more space in YARN containers for executor memory overhead than the 10% default set internally by Spark. The Amazon EMR default of 18.75% empirically showed fewer memory-related failures in TPC-DS benchmarks.
- Backported SPARK-26316 to improve performance.
In Amazon EMR version 5.19.0, 5.20.0, and 5.21.0, YARN node labels are stored in an HDFS directory. In some situations, this leads to core node startup delays and then causes cluster time-out and launch failure. Beginning with Amazon EMR 5.22.0, this issue is resolved. YARN node labels are stored on the local disk of each cluster node, avoiding dependencies on HDFS.

Known Issues

Hue (Fixed in Amazon EMR release version 5.24.0)
- Hue running on Amazon EMR does not support Solr. Beginning with Amazon EMR release version 5.20.0, a misconfiguration issue causes Solr to be enabled and a harmless error message to appear similar to the following:
  
  Solr server could not be contacted properly: HTTPConnectionPool('host=ip-xx-xx-xx-xx.ec2.internal', port=1978): Max retries exceeded with url: /solr/admin/info/system?user.name=hue&doAs=administrator&wt=json (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',))
  
  To prevent the Solr error message from appearing:
  1. Connect to the master node command line using SSH.
  2. Use a text editor to open the hue.ini file. For example:
    
    sudo vim /etc/hue/conf/hue.ini
  3. Search for the term "appblacklist" and modify the line to the following:
    
    appblacklist = search
  4. Save your changes and restart Hue as shown in the following example:
    
    sudo stop hue; sudo start hue

5.22.0 Component Versions

Component	Version	Description
aws-sagemaker-spark-sdk	1.2.1	Amazon SageMaker Spark SDK
emr-ddb	4.8.0	Amazon DynamoDB connector for Hadoop ecosystem applications.
emr-goodies	2.6.0	Extra convenience libraries for the Hadoop ecosystem.
emr-kinesis	3.4.0	Amazon Kinesis connector for Hadoop ecosystem applications.
emr-s3-dist-cp	2.11.0	Distributed copy application optimized for Amazon S3.
emr-s3-select	1.2.0	EMR S3Select Connector
emrfs	2.31.0	Amazon S3 connector for Hadoop ecosystem applications.
flink-client	1.7.1	Apache Flink command line client scripts and applications.
ganglia-monitor	3.7.2	Embedded Ganglia agent for Hadoop ecosystem applications along with the Ganglia monitoring agent.
ganglia-metadata-collector	3.7.2	Ganglia metadata collector for aggregating metrics from Ganglia monitoring agents.
ganglia-web	3.7.1	Web application for viewing metrics collected by the Ganglia metadata collector.
hadoop-client	2.8.5-amzn-2	Hadoop command-line clients such as 'hdfs', 'hadoop', or 'yarn'.
hadoop-hdfs-datanode	2.8.5-amzn-2	HDFS node-level service for storing blocks.
hadoop-hdfs-library	2.8.5-amzn-2	HDFS command-line client and library
hadoop-hdfs-namenode	2.8.5-amzn-2	HDFS service for tracking file names and block locations.
hadoop-httpfs-server	2.8.5-amzn-2	HTTP endpoint for HDFS operations.
hadoop-kms-server	2.8.5-amzn-2	Cryptographic key management server based on Hadoop's KeyProvider API.
hadoop-mapred	2.8.5-amzn-2	MapReduce execution engine libraries for running a MapReduce application.
hadoop-yarn-nodemanager	2.8.5-amzn-2	YARN service for managing containers on an individual node.
hadoop-yarn-resourcemanager	2.8.5-amzn-2	YARN service for allocating and managing cluster resources and distributed applications.
hadoop-yarn-timeline-server	2.8.5-amzn-2	Service for retrieving current and historical information for YARN applications.
hbase-hmaster	1.4.9	Service for an HBase cluster responsible for coordination of Regions and execution of administrative commands.
hbase-region-server	1.4.9	Service for serving one or more HBase regions.
hbase-client	1.4.9	HBase command-line client.
hbase-rest-server	1.4.9	Service providing a RESTful HTTP endpoint for HBase.
hbase-thrift-server	1.4.9	Service providing a Thrift endpoint to HBase.
hcatalog-client	2.3.4-amzn-1	The 'hcat' command line client for manipulating hcatalog-server.
hcatalog-server	2.3.4-amzn-1	Service providing HCatalog, a table and storage management layer for distributed applications.
hcatalog-webhcat-server	2.3.4-amzn-1	HTTP endpoint providing a REST interface to HCatalog.
hive-client	2.3.4-amzn-1	Hive command line client.
hive-hbase	2.3.4-amzn-1	Hive-hbase client.
hive-metastore-server	2.3.4-amzn-1	Service for accessing the Hive metastore, a semantic repository storing metadata for SQL on Hadoop operations.
hive-server2	2.3.4-amzn-1	Service for accepting Hive queries as web requests.
hue-server	4.3.0	Web application for analyzing data using Hadoop ecosystem applications
jupyterhub	0.9.4	Multi-user server for Jupyter notebooks
livy-server	0.5.0-incubating	REST interface for interacting with Apache Spark
nginx	1.12.1	nginx [engine x] is an HTTP and reverse proxy server
mahout-client	0.13.0	Library for machine learning.
mxnet	1.3.1	A flexible, scalable, and efficient library for deep learning.
mysql-server	5.5.54+	MySQL database server.
nvidia-cuda	9.2.88	Nvidia drivers and Cuda toolkit
oozie-client	5.1.0	Oozie command-line client.
oozie-server	5.1.0	Service for accepting Oozie workflow requests.
opencv	3.4.0	Open Source Computer Vision Library.
phoenix-library	4.14.1-HBase-1.4	The phoenix libraries for server and client
phoenix-query-server	4.14.1-HBase-1.4	A light weight server providing JDBC access as well as Protocol Buffers and JSON format access to the Avatica API
presto-coordinator	0.215	Service for accepting queries and managing query execution among presto-workers.
presto-worker	0.215	Service for executing pieces of a query.
pig-client	0.17.0	Pig command-line client.
r	3.4.1	The R Project for Statistical Computing
spark-client	2.4.0	Spark command-line clients.
spark-history-server	2.4.0	Web UI for viewing logged events for the lifetime of a completed Spark application.
spark-on-yarn	2.4.0	In-memory execution engine for YARN.
spark-yarn-slave	2.4.0	Apache Spark libraries needed by YARN slaves.
sqoop-client	1.4.7	Apache Sqoop command-line client.
tensorflow	1.12.0	TensorFlow open source software library for high performance numerical computation.
tez-on-yarn	0.9.1	The tez YARN application and libraries.
webserver	2.4.25+	Apache HTTP server.
zeppelin-server	0.8.1	Web-based notebook that enables interactive data analytics.
zookeeper-server	3.4.13	Centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
zookeeper-client	3.4.13	ZooKeeper command line client.

5.22.0 Configuration Classifications

emr-5.22.0 Classifications

Classifications	Description
capacity-scheduler	Change values in Hadoop's capacity-scheduler.xml file.
container-log4j	Change values in Hadoop YARN's container-log4j.properties file.
core-site	Change values in Hadoop's core-site.xml file.
emrfs-site	Change EMRFS settings.
flink-conf	Change flink-conf.yaml settings.
flink-log4j	Change Flink log4j.properties settings.
flink-log4j-yarn-session	Change Flink log4j-yarn-session.properties settings.
flink-log4j-cli	Change Flink log4j-cli.properties settings.
hadoop-env	Change values in the Hadoop environment for all Hadoop components.
hadoop-log4j	Change values in Hadoop's log4j.properties file.
hadoop-ssl-server	Change hadoop ssl server configuration
hadoop-ssl-client	Change hadoop ssl client configuration
hbase	Amazon EMR-curated settings for Apache HBase.
hbase-env	Change values in HBase's environment.
hbase-log4j	Change values in HBase's hbase-log4j.properties file.
hbase-metrics	Change values in HBase's hadoop-metrics2-hbase.properties file.
hbase-policy	Change values in HBase's hbase-policy.xml file.
hbase-site	Change values in HBase's hbase-site.xml file.
hdfs-encryption-zones	Configure HDFS encryption zones.
hdfs-site	Change values in HDFS's hdfs-site.xml.
hcatalog-env	Change values in HCatalog's environment.
hcatalog-server-jndi	Change values in HCatalog's jndi.properties.
hcatalog-server-proto-hive-site	Change values in HCatalog's proto-hive-site.xml.
hcatalog-webhcat-env	Change values in HCatalog WebHCat's environment.
hcatalog-webhcat-log4j2	Change values in HCatalog WebHCat's log4j2.properties.
hcatalog-webhcat-site	Change values in HCatalog WebHCat's webhcat-site.xml file.
hive-beeline-log4j2	Change values in Hive's beeline-log4j2.properties file.
hive-parquet-logging	Change values in Hive's parquet-logging.properties file.
hive-env	Change values in the Hive environment.
hive-exec-log4j2	Change values in Hive's hive-exec-log4j2.properties file.
hive-llap-daemon-log4j2	Change values in Hive's llap-daemon-log4j2.properties file.
hive-log4j2	Change values in Hive's hive-log4j2.properties file.
hive-site	Change values in Hive's hive-site.xml file
hiveserver2-site	Change values in Hive Server2's hiveserver2-site.xml file
hue-ini	Change values in Hue's ini file
httpfs-env	Change values in the HTTPFS environment.
httpfs-site	Change values in Hadoop's httpfs-site.xml file.
hadoop-kms-acls	Change values in Hadoop's kms-acls.xml file.
hadoop-kms-env	Change values in the Hadoop KMS environment.
hadoop-kms-log4j	Change values in Hadoop's kms-log4j.properties file.
hadoop-kms-site	Change values in Hadoop's kms-site.xml file.
jupyter-notebook-conf	Change values in Jupyter Notebook's jupyter_notebook_config.py file.
jupyter-hub-conf	Change values in JupyterHubs's jupyterhub_config.py file.
jupyter-s3-conf	Configure Jupyter Notebook S3 persistence.
jupyter-sparkmagic-conf	Change values in Sparkmagic's config.json file.
livy-conf	Change values in Livy's livy.conf file.
livy-env	Change values in the Livy environment.
livy-log4j	Change Livy log4j.properties settings.
mapred-env	Change values in the MapReduce application's environment.
mapred-site	Change values in the MapReduce application's mapred-site.xml file.
oozie-env	Change values in Oozie's environment.
oozie-log4j	Change values in Oozie's oozie-log4j.properties file.
oozie-site	Change values in Oozie's oozie-site.xml file.
phoenix-hbase-metrics	Change values in Phoenix's hadoop-metrics2-hbase.properties file.
phoenix-hbase-site	Change values in Phoenix's hbase-site.xml file.
phoenix-log4j	Change values in Phoenix's log4j.properties file.
phoenix-metrics	Change values in Phoenix's hadoop-metrics2-phoenix.properties file.
pig-env	Change values in the Pig environment.
pig-properties	Change values in Pig's pig.properties file.
pig-log4j	Change values in Pig's log4j.properties file.
presto-log	Change values in Presto's log.properties file.
presto-config	Change values in Presto's config.properties file.
presto-password-authenticator	Change values in Presto's password-authenticator.properties file.
presto-env	Change values in Presto's presto-env.sh file.
presto-node	Change values in Presto's node.properties file.
presto-connector-blackhole	Change values in Presto's blackhole.properties file.
presto-connector-cassandra	Change values in Presto's cassandra.properties file.
presto-connector-hive	Change values in Presto's hive.properties file.
presto-connector-jmx	Change values in Presto's jmx.properties file.
presto-connector-kafka	Change values in Presto's kafka.properties file.
presto-connector-localfile	Change values in Presto's localfile.properties file.
presto-connector-memory	Change values in Presto's memory.properties file.
presto-connector-mongodb	Change values in Presto's mongodb.properties file.
presto-connector-mysql	Change values in Presto's mysql.properties file.
presto-connector-postgresql	Change values in Presto's postgresql.properties file.
presto-connector-raptor	Change values in Presto's raptor.properties file.
presto-connector-redis	Change values in Presto's redis.properties file.
presto-connector-redshift	Change values in Presto's redshift.properties file.
presto-connector-tpch	Change values in Presto's tpch.properties file.
presto-connector-tpcds	Change values in Presto's tpcds.properties file.
spark	Amazon EMR-curated settings for Apache Spark.
spark-defaults	Change values in Spark's spark-defaults.conf file.
spark-env	Change values in the Spark environment.
spark-hive-site	Change values in Spark's hive-site.xml file
spark-log4j	Change values in Spark's log4j.properties file.
spark-metrics	Change values in Spark's metrics.properties file.
sqoop-env	Change values in Sqoop's environment.
sqoop-oraoop-site	Change values in Sqoop OraOop's oraoop-site.xml file.
sqoop-site	Change values in Sqoop's sqoop-site.xml file.
tez-site	Change values in Tez's tez-site.xml file.
yarn-env	Change values in the YARN environment.
yarn-site	Change values in YARN's yarn-site.xml file.
zeppelin-env	Change values in the Zeppelin environment.
zookeeper-config	Change values in ZooKeeper's zoo.cfg file.
zookeeper-log4j	Change values in ZooKeeper's log4j.properties file.

5.21.x

There are multiple releases within the 5.21 series. Choose a link below to see information for a specific release within this tab.

5.21.1 (Latest) | 5.21.0

Amazon EMR Release 5.21.1

Release 5.21.1 Application Versions

The diagram below depicts the application versions available in this release of Amazon EMR and the application versions in the preceding four Amazon EMR releases.

For a comprehensive history of application versions for each release of Amazon EMR, see the following diagrams:

Release 5.21.1 Release Notes

The following release notes include information for Amazon EMR release version 5.21.1. Changes are relative to 5.21.0.

Initial release date: July 18, 2019

Changes, Enhancements, and Resolved Issues

Updated the default Amazon Linux AMI for EMR to include important Linux kernel security updates, including the TCP SACK Denial of Service Issue (AWS-2019-005).

Release 5.21.1 Component Versions

Component	Version	Description
aws-sagemaker-spark-sdk	1.2.1	Amazon SageMaker Spark SDK
emr-ddb	4.7.0	Amazon DynamoDB connector for Hadoop ecosystem applications.
emr-goodies	2.5.1	Extra convenience libraries for the Hadoop ecosystem.
emr-kinesis	3.4.0	Amazon Kinesis connector for Hadoop ecosystem applications.
emr-s3-dist-cp	2.11.0	Distributed copy application optimized for Amazon S3.
emr-s3-select	1.2.0	EMR S3Select Connector
emrfs	2.30.0	Amazon S3 connector for Hadoop ecosystem applications.
flink-client	1.7.0	Apache Flink command line client scripts and applications.
ganglia-monitor	3.7.2	Embedded Ganglia agent for Hadoop ecosystem applications along with the Ganglia monitoring agent.
ganglia-metadata-collector	3.7.2	Ganglia metadata collector for aggregating metrics from Ganglia monitoring agents.
ganglia-web	3.7.1	Web application for viewing metrics collected by the Ganglia metadata collector.
hadoop-client	2.8.5-amzn-1	Hadoop command-line clients such as 'hdfs', 'hadoop', or 'yarn'.
hadoop-hdfs-datanode	2.8.5-amzn-1	HDFS node-level service for storing blocks.
hadoop-hdfs-library	2.8.5-amzn-1	HDFS command-line client and library
hadoop-hdfs-namenode	2.8.5-amzn-1	HDFS service for tracking file names and block locations.
hadoop-httpfs-server	2.8.5-amzn-1	HTTP endpoint for HDFS operations.
hadoop-kms-server	2.8.5-amzn-1	Cryptographic key management server based on Hadoop's KeyProvider API.
hadoop-mapred	2.8.5-amzn-1	MapReduce execution engine libraries for running a MapReduce application.
hadoop-yarn-nodemanager	2.8.5-amzn-1	YARN service for managing containers on an individual node.
hadoop-yarn-resourcemanager	2.8.5-amzn-1	YARN service for allocating and managing cluster resources and distributed applications.
hadoop-yarn-timeline-server	2.8.5-amzn-1	Service for retrieving current and historical information for YARN applications.
hbase-hmaster	1.4.8	Service for an HBase cluster responsible for coordination of Regions and execution of administrative commands.
hbase-region-server	1.4.8	Service for serving one or more HBase regions.
hbase-client	1.4.8	HBase command-line client.
hbase-rest-server	1.4.8	Service providing a RESTful HTTP endpoint for HBase.
hbase-thrift-server	1.4.8	Service providing a Thrift endpoint to HBase.
hcatalog-client	2.3.4-amzn-0	The 'hcat' command line client for manipulating hcatalog-server.
hcatalog-server	2.3.4-amzn-0	Service providing HCatalog, a table and storage management layer for distributed applications.
hcatalog-webhcat-server	2.3.4-amzn-0	HTTP endpoint providing a REST interface to HCatalog.
hive-client	2.3.4-amzn-0	Hive command line client.
hive-hbase	2.3.4-amzn-0	Hive-hbase client.
hive-metastore-server	2.3.4-amzn-0	Service for accessing the Hive metastore, a semantic repository storing metadata for SQL on Hadoop operations.
hive-server2	2.3.4-amzn-0	Service for accepting Hive queries as web requests.
hue-server	4.3.0	Web application for analyzing data using Hadoop ecosystem applications
jupyterhub	0.9.4	Multi-user server for Jupyter notebooks
livy-server	0.5.0-incubating	REST interface for interacting with Apache Spark
nginx	1.12.1	nginx [engine x] is an HTTP and reverse proxy server
mahout-client	0.13.0	Library for machine learning.
mxnet	1.3.1	A flexible, scalable, and efficient library for deep learning.
mysql-server	5.5.54+	MySQL database server.
nvidia-cuda	9.2.88	Nvidia drivers and Cuda toolkit
oozie-client	5.0.0	Oozie command-line client.
oozie-server	5.0.0	Service for accepting Oozie workflow requests.
opencv	3.4.0	Open Source Computer Vision Library.
phoenix-library	4.14.0-HBase-1.4	The phoenix libraries for server and client
phoenix-query-server	4.14.0-HBase-1.4	A light weight server providing JDBC access as well as Protocol Buffers and JSON format access to the Avatica API
presto-coordinator	0.215	Service for accepting queries and managing query execution among presto-workers.
presto-worker	0.215	Service for executing pieces of a query.
pig-client	0.17.0	Pig command-line client.
r	3.4.1	The R Project for Statistical Computing
spark-client	2.4.0	Spark command-line clients.
spark-history-server	2.4.0	Web UI for viewing logged events for the lifetime of a completed Spark application.
spark-on-yarn	2.4.0	In-memory execution engine for YARN.
spark-yarn-slave	2.4.0	Apache Spark libraries needed by YARN slaves.
sqoop-client	1.4.7	Apache Sqoop command-line client.
tensorflow	1.12.0	TensorFlow open source software library for high performance numerical computation.
tez-on-yarn	0.9.1	The tez YARN application and libraries.
webserver	2.4.25+	Apache HTTP server.
zeppelin-server	0.8.0	Web-based notebook that enables interactive data analytics.
zookeeper-server	3.4.13	Centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
zookeeper-client	3.4.13	ZooKeeper command line client.

Release 5.21.1 Configuration Classifications

emr-5.21.1 Classifications

Classifications	Description
capacity-scheduler	Change values in Hadoop's capacity-scheduler.xml file.
container-log4j	Change values in Hadoop YARN's container-log4j.properties file.
core-site	Change values in Hadoop's core-site.xml file.
emrfs-site	Change EMRFS settings.
flink-conf	Change flink-conf.yaml settings.
flink-log4j	Change Flink log4j.properties settings.
flink-log4j-yarn-session	Change Flink log4j-yarn-session.properties settings.
flink-log4j-cli	Change Flink log4j-cli.properties settings.
hadoop-env	Change values in the Hadoop environment for all Hadoop components.
hadoop-log4j	Change values in Hadoop's log4j.properties file.
hadoop-ssl-server	Change hadoop ssl server configuration
hadoop-ssl-client	Change hadoop ssl client configuration
hbase	Amazon EMR-curated settings for Apache HBase.
hbase-env	Change values in HBase's environment.
hbase-log4j	Change values in HBase's hbase-log4j.properties file.
hbase-metrics	Change values in HBase's hadoop-metrics2-hbase.properties file.
hbase-policy	Change values in HBase's hbase-policy.xml file.
hbase-site	Change values in HBase's hbase-site.xml file.
hdfs-encryption-zones	Configure HDFS encryption zones.
hdfs-site	Change values in HDFS's hdfs-site.xml.
hcatalog-env	Change values in HCatalog's environment.
hcatalog-server-jndi	Change values in HCatalog's jndi.properties.
hcatalog-server-proto-hive-site	Change values in HCatalog's proto-hive-site.xml.
hcatalog-webhcat-env	Change values in HCatalog WebHCat's environment.
hcatalog-webhcat-log4j2	Change values in HCatalog WebHCat's log4j2.properties.
hcatalog-webhcat-site	Change values in HCatalog WebHCat's webhcat-site.xml file.
hive-beeline-log4j2	Change values in Hive's beeline-log4j2.properties file.
hive-parquet-logging	Change values in Hive's parquet-logging.properties file.
hive-env	Change values in the Hive environment.
hive-exec-log4j2	Change values in Hive's hive-exec-log4j2.properties file.
hive-llap-daemon-log4j2	Change values in Hive's llap-daemon-log4j2.properties file.
hive-log4j2	Change values in Hive's hive-log4j2.properties file.
hive-site	Change values in Hive's hive-site.xml file
hiveserver2-site	Change values in Hive Server2's hiveserver2-site.xml file
hue-ini	Change values in Hue's ini file
httpfs-env	Change values in the HTTPFS environment.
httpfs-site	Change values in Hadoop's httpfs-site.xml file.
hadoop-kms-acls	Change values in Hadoop's kms-acls.xml file.
hadoop-kms-env	Change values in the Hadoop KMS environment.
hadoop-kms-log4j	Change values in Hadoop's kms-log4j.properties file.
hadoop-kms-site	Change values in Hadoop's kms-site.xml file.
jupyter-notebook-conf	Change values in Jupyter Notebook's jupyter_notebook_config.py file.
jupyter-hub-conf	Change values in JupyterHubs's jupyterhub_config.py file.
jupyter-s3-conf	Configure Jupyter Notebook S3 persistence.
jupyter-sparkmagic-conf	Change values in Sparkmagic's config.json file.
livy-conf	Change values in Livy's livy.conf file.
livy-env	Change values in the Livy environment.
livy-log4j	Change Livy log4j.properties settings.
mapred-env	Change values in the MapReduce application's environment.
mapred-site	Change values in the MapReduce application's mapred-site.xml file.
oozie-env	Change values in Oozie's environment.
oozie-log4j	Change values in Oozie's oozie-log4j.properties file.
oozie-site	Change values in Oozie's oozie-site.xml file.
phoenix-hbase-metrics	Change values in Phoenix's hadoop-metrics2-hbase.properties file.
phoenix-hbase-site	Change values in Phoenix's hbase-site.xml file.
phoenix-log4j	Change values in Phoenix's log4j.properties file.
phoenix-metrics	Change values in Phoenix's hadoop-metrics2-phoenix.properties file.
pig-env	Change values in the Pig environment.
pig-properties	Change values in Pig's pig.properties file.
pig-log4j	Change values in Pig's log4j.properties file.
presto-log	Change values in Presto's log.properties file.
presto-config	Change values in Presto's config.properties file.
presto-password-authenticator	Change values in Presto's password-authenticator.properties file.
presto-env	Change values in Presto's presto-env.sh file.
presto-node	Change values in Presto's node.properties file.
presto-connector-blackhole	Change values in Presto's blackhole.properties file.
presto-connector-cassandra	Change values in Presto's cassandra.properties file.
presto-connector-hive	Change values in Presto's hive.properties file.
presto-connector-jmx	Change values in Presto's jmx.properties file.
presto-connector-kafka	Change values in Presto's kafka.properties file.
presto-connector-localfile	Change values in Presto's localfile.properties file.
presto-connector-memory	Change values in Presto's memory.properties file.
presto-connector-mongodb	Change values in Presto's mongodb.properties file.
presto-connector-mysql	Change values in Presto's mysql.properties file.
presto-connector-postgresql	Change values in Presto's postgresql.properties file.
presto-connector-raptor	Change values in Presto's raptor.properties file.
presto-connector-redis	Change values in Presto's redis.properties file.
presto-connector-redshift	Change values in Presto's redshift.properties file.
presto-connector-tpch	Change values in Presto's tpch.properties file.
presto-connector-tpcds	Change values in Presto's tpcds.properties file.
spark	Amazon EMR-curated settings for Apache Spark.
spark-defaults	Change values in Spark's spark-defaults.conf file.
spark-env	Change values in the Spark environment.
spark-hive-site	Change values in Spark's hive-site.xml file
spark-log4j	Change values in Spark's log4j.properties file.
spark-metrics	Change values in Spark's metrics.properties file.
sqoop-env	Change values in Sqoop's environment.
sqoop-oraoop-site	Change values in Sqoop OraOop's oraoop-site.xml file.
sqoop-site	Change values in Sqoop's sqoop-site.xml file.
tez-site	Change values in Tez's tez-site.xml file.
yarn-env	Change values in the YARN environment.
yarn-site	Change values in YARN's yarn-site.xml file.
zeppelin-env	Change values in the Zeppelin environment.
zookeeper-config	Change values in ZooKeeper's zoo.cfg file.
zookeeper-log4j	Change values in ZooKeeper's log4j.properties file.

Amazon EMR Release 5.21.0

Release 5.21.0 Application Versions

The diagram below depicts the application versions available in this release of Amazon EMR and the application versions in the preceding four Amazon EMR releases.

For a comprehensive history of application versions for each release of Amazon EMR, see the following diagrams:

Release 5.21.0 Release Notes

The following release notes include information for Amazon EMR release version 5.21.0. Changes are relative to 5.20.0.

Initial release date: February 18, 2019

Last updated date: April 3, 2019

Upgrades

Flink 1.7.0
Presto 0.215
AWS SDK for Java 1.11.479

New Features

(April 3, 2019) With Amazon EMR version 5.21.0 and later, you can override cluster configurations and specify additional configuration classifications for each instance group in a running cluster. You do this by using the Amazon EMR console, the AWS Command Line Interface (AWS CLI), or the AWS SDK. For more information, see Supplying a Configuration for an Instance Group in a Running Cluster.

Changes, Enhancements, and Resolved Issues

Zeppelin
- Backported ZEPPELIN-3878.

Known Issues

Hue (Fixed in Amazon EMR release version 5.24.0)
- Hue running on Amazon EMR does not support Solr. Beginning with Amazon EMR release version 5.20.0, a misconfiguration issue causes Solr to be enabled and a harmless error message to appear similar to the following:
  
  Solr server could not be contacted properly: HTTPConnectionPool('host=ip-xx-xx-xx-xx.ec2.internal', port=1978): Max retries exceeded with url: /solr/admin/info/system?user.name=hue&doAs=administrator&wt=json (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',))
  
  To prevent the Solr error message from appearing:
  1. Connect to the master node command line using SSH.
  2. Use a text editor to open the hue.ini file. For example:
    
    sudo vim /etc/hue/conf/hue.ini
  3. Search for the term "appblacklist" and modify the line to the following:
    
    appblacklist = search
  4. Save your changes and restart Hue as shown in the following example:
    
    sudo stop hue; sudo start hue
Tez
- This issue was fixed in Amazon EMR 5.22.0.
  
  When you connect to the Tez UI at http://MasterDNS:8080/tez-ui through an SSH connection to the cluster master node, the error "Adapter operation failed - Timeline server (ATS) is out of reach. Either it is down, or CORS is not enabled" appears, or tasks unexpectedly show N/A.
  
  This is caused by the Tez UI making requests to the YARN Timeline Server using localhost rather than the host name of the master node. As a workaround, a script is available to run as a bootstrap action or step. The script updates the host name in the Tez configs.env file. For more information and the location of the script, see the Bootstrap Instructions.
In Amazon EMR version 5.19.0, 5.20.0, and 5.21.0, YARN node labels are stored in an HDFS directory. In some situations, this leads to core node startup delays and then causes cluster time-out and launch failure. Beginning with Amazon EMR 5.22.0, this issue is resolved. YARN node labels are stored on the local disk of each cluster node, avoiding dependencies on HDFS.

Release 5.21.0 Component Versions

Component	Version	Description
aws-sagemaker-spark-sdk	1.2.1	Amazon SageMaker Spark SDK
emr-ddb	4.7.0	Amazon DynamoDB connector for Hadoop ecosystem applications.
emr-goodies	2.5.1	Extra convenience libraries for the Hadoop ecosystem.
emr-kinesis	3.4.0	Amazon Kinesis connector for Hadoop ecosystem applications.
emr-s3-dist-cp	2.11.0	Distributed copy application optimized for Amazon S3.
emr-s3-select	1.2.0	EMR S3Select Connector
emrfs	2.30.0	Amazon S3 connector for Hadoop ecosystem applications.
flink-client	1.7.0	Apache Flink command line client scripts and applications.
ganglia-monitor	3.7.2	Embedded Ganglia agent for Hadoop ecosystem applications along with the Ganglia monitoring agent.
ganglia-metadata-collector	3.7.2	Ganglia metadata collector for aggregating metrics from Ganglia monitoring agents.
ganglia-web	3.7.1	Web application for viewing metrics collected by the Ganglia metadata collector.
hadoop-client	2.8.5-amzn-1	Hadoop command-line clients such as 'hdfs', 'hadoop', or 'yarn'.
hadoop-hdfs-datanode	2.8.5-amzn-1	HDFS node-level service for storing blocks.
hadoop-hdfs-library	2.8.5-amzn-1	HDFS command-line client and library
hadoop-hdfs-namenode	2.8.5-amzn-1	HDFS service for tracking file names and block locations.
hadoop-httpfs-server	2.8.5-amzn-1	HTTP endpoint for HDFS operations.
hadoop-kms-server	2.8.5-amzn-1	Cryptographic key management server based on Hadoop's KeyProvider API.
hadoop-mapred	2.8.5-amzn-1	MapReduce execution engine libraries for running a MapReduce application.
hadoop-yarn-nodemanager	2.8.5-amzn-1	YARN service for managing containers on an individual node.
hadoop-yarn-resourcemanager	2.8.5-amzn-1	YARN service for allocating and managing cluster resources and distributed applications.
hadoop-yarn-timeline-server	2.8.5-amzn-1	Service for retrieving current and historical information for YARN applications.
hbase-hmaster	1.4.8	Service for an HBase cluster responsible for coordination of Regions and execution of administrative commands.
hbase-region-server	1.4.8	Service for serving one or more HBase regions.
hbase-client	1.4.8	HBase command-line client.
hbase-rest-server	1.4.8	Service providing a RESTful HTTP endpoint for HBase.
hbase-thrift-server	1.4.8	Service providing a Thrift endpoint to HBase.
hcatalog-client	2.3.4-amzn-0	The 'hcat' command line client for manipulating hcatalog-server.
hcatalog-server	2.3.4-amzn-0	Service providing HCatalog, a table and storage management layer for distributed applications.
hcatalog-webhcat-server	2.3.4-amzn-0	HTTP endpoint providing a REST interface to HCatalog.
hive-client	2.3.4-amzn-0	Hive command line client.
hive-hbase	2.3.4-amzn-0	Hive-hbase client.
hive-metastore-server	2.3.4-amzn-0	Service for accessing the Hive metastore, a semantic repository storing metadata for SQL on Hadoop operations.
hive-server2	2.3.4-amzn-0	Service for accepting Hive queries as web requests.
hue-server	4.3.0	Web application for analyzing data using Hadoop ecosystem applications
jupyterhub	0.9.4	Multi-user server for Jupyter notebooks
livy-server	0.5.0-incubating	REST interface for interacting with Apache Spark
nginx	1.12.1	nginx [engine x] is an HTTP and reverse proxy server
mahout-client	0.13.0	Library for machine learning.
mxnet	1.3.1	A flexible, scalable, and efficient library for deep learning.
mysql-server	5.5.54+	MySQL database server.
nvidia-cuda	9.2.88	Nvidia drivers and Cuda toolkit
oozie-client	5.0.0	Oozie command-line client.
oozie-server	5.0.0	Service for accepting Oozie workflow requests.
opencv	3.4.0	Open Source Computer Vision Library.
phoenix-library	4.14.0-HBase-1.4	The phoenix libraries for server and client
phoenix-query-server	4.14.0-HBase-1.4	A light weight server providing JDBC access as well as Protocol Buffers and JSON format access to the Avatica API
presto-coordinator	0.215	Service for accepting queries and managing query execution among presto-workers.
presto-worker	0.215	Service for executing pieces of a query.
pig-client	0.17.0	Pig command-line client.
r	3.4.1	The R Project for Statistical Computing
spark-client	2.4.0	Spark command-line clients.
spark-history-server	2.4.0	Web UI for viewing logged events for the lifetime of a completed Spark application.
spark-on-yarn	2.4.0	In-memory execution engine for YARN.
spark-yarn-slave	2.4.0	Apache Spark libraries needed by YARN slaves.
sqoop-client	1.4.7	Apache Sqoop command-line client.
tensorflow	1.12.0	TensorFlow open source software library for high performance numerical computation.
tez-on-yarn	0.9.1	The tez YARN application and libraries.
webserver	2.4.25+	Apache HTTP server.
zeppelin-server	0.8.0	Web-based notebook that enables interactive data analytics.
zookeeper-server	3.4.13	Centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
zookeeper-client	3.4.13	ZooKeeper command line client.

Release 5.21.0 Configuration Classifications

emr-5.21.0 Classifications

Classifications	Description
capacity-scheduler	Change values in Hadoop's capacity-scheduler.xml file.
container-log4j	Change values in Hadoop YARN's container-log4j.properties file.
core-site	Change values in Hadoop's core-site.xml file.
emrfs-site	Change EMRFS settings.
flink-conf	Change flink-conf.yaml settings.
flink-log4j	Change Flink log4j.properties settings.
flink-log4j-yarn-session	Change Flink log4j-yarn-session.properties settings.
flink-log4j-cli	Change Flink log4j-cli.properties settings.
hadoop-env	Change values in the Hadoop environment for all Hadoop components.
hadoop-log4j	Change values in Hadoop's log4j.properties file.
hadoop-ssl-server	Change hadoop ssl server configuration
hadoop-ssl-client	Change hadoop ssl client configuration
hbase	Amazon EMR-curated settings for Apache HBase.
hbase-env	Change values in HBase's environment.
hbase-log4j	Change values in HBase's hbase-log4j.properties file.
hbase-metrics	Change values in HBase's hadoop-metrics2-hbase.properties file.
hbase-policy	Change values in HBase's hbase-policy.xml file.
hbase-site	Change values in HBase's hbase-site.xml file.
hdfs-encryption-zones	Configure HDFS encryption zones.
hdfs-site	Change values in HDFS's hdfs-site.xml.
hcatalog-env	Change values in HCatalog's environment.
hcatalog-server-jndi	Change values in HCatalog's jndi.properties.
hcatalog-server-proto-hive-site	Change values in HCatalog's proto-hive-site.xml.
hcatalog-webhcat-env	Change values in HCatalog WebHCat's environment.
hcatalog-webhcat-log4j2	Change values in HCatalog WebHCat's log4j2.properties.
hcatalog-webhcat-site	Change values in HCatalog WebHCat's webhcat-site.xml file.
hive-beeline-log4j2	Change values in Hive's beeline-log4j2.properties file.
hive-parquet-logging	Change values in Hive's parquet-logging.properties file.
hive-env	Change values in the Hive environment.
hive-exec-log4j2	Change values in Hive's hive-exec-log4j2.properties file.
hive-llap-daemon-log4j2	Change values in Hive's llap-daemon-log4j2.properties file.
hive-log4j2	Change values in Hive's hive-log4j2.properties file.
hive-site	Change values in Hive's hive-site.xml file
hiveserver2-site	Change values in Hive Server2's hiveserver2-site.xml file
hue-ini	Change values in Hue's ini file
httpfs-env	Change values in the HTTPFS environment.
httpfs-site	Change values in Hadoop's httpfs-site.xml file.
hadoop-kms-acls	Change values in Hadoop's kms-acls.xml file.
hadoop-kms-env	Change values in the Hadoop KMS environment.
hadoop-kms-log4j	Change values in Hadoop's kms-log4j.properties file.
hadoop-kms-site	Change values in Hadoop's kms-site.xml file.
jupyter-notebook-conf	Change values in Jupyter Notebook's jupyter_notebook_config.py file.
jupyter-hub-conf	Change values in JupyterHubs's jupyterhub_config.py file.
jupyter-s3-conf	Configure Jupyter Notebook S3 persistence.
jupyter-sparkmagic-conf	Change values in Sparkmagic's config.json file.
livy-conf	Change values in Livy's livy.conf file.
livy-env	Change values in the Livy environment.
livy-log4j	Change Livy log4j.properties settings.
mapred-env	Change values in the MapReduce application's environment.
mapred-site	Change values in the MapReduce application's mapred-site.xml file.
oozie-env	Change values in Oozie's environment.
oozie-log4j	Change values in Oozie's oozie-log4j.properties file.
oozie-site	Change values in Oozie's oozie-site.xml file.
phoenix-hbase-metrics	Change values in Phoenix's hadoop-metrics2-hbase.properties file.
phoenix-hbase-site	Change values in Phoenix's hbase-site.xml file.
phoenix-log4j	Change values in Phoenix's log4j.properties file.
phoenix-metrics	Change values in Phoenix's hadoop-metrics2-phoenix.properties file.
pig-env	Change values in the Pig environment.
pig-properties	Change values in Pig's pig.properties file.
pig-log4j	Change values in Pig's log4j.properties file.
presto-log	Change values in Presto's log.properties file.
presto-config	Change values in Presto's config.properties file.
presto-password-authenticator	Change values in Presto's password-authenticator.properties file.
presto-env	Change values in Presto's presto-env.sh file.
presto-node	Change values in Presto's node.properties file.
presto-connector-blackhole	Change values in Presto's blackhole.properties file.
presto-connector-cassandra	Change values in Presto's cassandra.properties file.
presto-connector-hive	Change values in Presto's hive.properties file.
presto-connector-jmx	Change values in Presto's jmx.properties file.
presto-connector-kafka	Change values in Presto's kafka.properties file.
presto-connector-localfile	Change values in Presto's localfile.properties file.
presto-connector-memory	Change values in Presto's memory.properties file.
presto-connector-mongodb	Change values in Presto's mongodb.properties file.
presto-connector-mysql	Change values in Presto's mysql.properties file.
presto-connector-postgresql	Change values in Presto's postgresql.properties file.
presto-connector-raptor	Change values in Presto's raptor.properties file.
presto-connector-redis	Change values in Presto's redis.properties file.
presto-connector-redshift	Change values in Presto's redshift.properties file.
presto-connector-tpch	Change values in Presto's tpch.properties file.
presto-connector-tpcds	Change values in Presto's tpcds.properties file.
spark	Amazon EMR-curated settings for Apache Spark.
spark-defaults	Change values in Spark's spark-defaults.conf file.
spark-env	Change values in the Spark environment.
spark-hive-site	Change values in Spark's hive-site.xml file
spark-log4j	Change values in Spark's log4j.properties file.
spark-metrics	Change values in Spark's metrics.properties file.
sqoop-env	Change values in Sqoop's environment.
sqoop-oraoop-site	Change values in Sqoop OraOop's oraoop-site.xml file.
sqoop-site	Change values in Sqoop's sqoop-site.xml file.
tez-site	Change values in Tez's tez-site.xml file.
yarn-env	Change values in the YARN environment.
yarn-site	Change values in YARN's yarn-site.xml file.
zeppelin-env	Change values in the Zeppelin environment.
zookeeper-config	Change values in ZooKeeper's zoo.cfg file.
zookeeper-log4j	Change values in ZooKeeper's log4j.properties file.

5.20.0

Amazon EMR Release 5.20.0

5.20.0 Application Versions

The diagram below depicts the application versions available in this release of Amazon EMR and the application versions in the preceding four Amazon EMR releases.

For a comprehensive history of application versions for each release of Amazon EMR, see the following diagrams:

5.20.0 Release Notes

The following release notes include information for Amazon EMR release version 5.20.0. Changes are relative to 5.19.0.

Initial release date: December 18, 2018

Last updated date: January 22, 2019

Upgrades

Flink 1.6.2
HBase 1.4.8
Hive 2.3.4
Hue 4.3.0
MXNet 1.3.1
Presto 0.214
Spark 2.4.0
TensorFlow 1.12.0
Tez 0.9.1
AWS SDK for Java 1.11.461

New Features

(January 22, 2019) Kerberos in Amazon EMR has been improved to support authenticating principals from an external KDC. This centralizes principal management because multiple clusters can share a single, external KDC. In addition, the external KDC can have a cross-realm trust with an Active Directory domain. This allows all clusters to authenticate principals from Active Directory. For more information, see Use Kerberos Authentication in the Amazon EMR Management Guide.

Changes, Enhancements, and Resolved Issues

Default Amazon Linux AMI for Amazon EMR
- Python3 package was upgraded from python 3.4 to 3.6.
The EMRFS S3-optimized committer
- The EMRFS S3-optimized committer is now enabled by default, which improves write performance. For more information, see Using the EMRFS S3-optimized Committer.
Hive
- Backported HIVE-16686.
Glue with Spark and Hive
- In EMR 5.20.0 or later, parallel partition pruning is enabled automatically for Spark and Hive when AWS Glue Data Catalog is used as the metastore. This change significantly reduces query planning time by executing multiple requests in parallel to retrieve partitions. The total number of segments that can be executed concurrently range between 1 and 10. The default value is 5, which is a recommended setting. You can change it by specifying the property aws.glue.partition.num.segments in hive-site configuration classification. If throttling occurs, you can turn off the feature by changing the value to 1. For more information, see AWS Glue Segment Structure.

Known Issues

Hue (Fixed in Amazon EMR release version 5.24.0)
- Hue running on Amazon EMR does not support Solr. Beginning with Amazon EMR release version 5.20.0, a misconfiguration issue causes Solr to be enabled and a harmless error message to appear similar to the following:
  
  Solr server could not be contacted properly: HTTPConnectionPool('host=ip-xx-xx-xx-xx.ec2.internal', port=1978): Max retries exceeded with url: /solr/admin/info/system?user.name=hue&doAs=administrator&wt=json (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',))
  
  To prevent the Solr error message from appearing:
  1. Connect to the master node command line using SSH.
  2. Use a text editor to open the hue.ini file. For example:
    
    sudo vim /etc/hue/conf/hue.ini
  3. Search for the term "appblacklist" and modify the line to the following:
    
    appblacklist = search
  4. Save your changes and restart Hue as shown in the following example:
    
    sudo stop hue; sudo start hue
Tez
- This issue was fixed in Amazon EMR 5.22.0.
  
  When you connect to the Tez UI at http://MasterDNS:8080/tez-ui through an SSH connection to the cluster master node, the error "Adapter operation failed - Timeline server (ATS) is out of reach. Either it is down, or CORS is not enabled" appears, or tasks unexpectedly show N/A.
  
  This is caused by the Tez UI making requests to the YARN Timeline Server using localhost rather than the host name of the master node. As a workaround, a script is available to run as a bootstrap action or step. The script updates the host name in the Tez configs.env file. For more information and the location of the script, see the Bootstrap Instructions.
In Amazon EMR version 5.19.0, 5.20.0, and 5.21.0, YARN node labels are stored in an HDFS directory. In some situations, this leads to core node startup delays and then causes cluster time-out and launch failure. Beginning with Amazon EMR 5.22.0, this issue is resolved. YARN node labels are stored on the local disk of each cluster node, avoiding dependencies on HDFS.

5.20.0 Component Versions

Component	Version	Description
aws-sagemaker-spark-sdk	1.2.1	Amazon SageMaker Spark SDK
emr-ddb	4.7.0	Amazon DynamoDB connector for Hadoop ecosystem applications.
emr-goodies	2.5.1	Extra convenience libraries for the Hadoop ecosystem.
emr-kinesis	3.4.0	Amazon Kinesis connector for Hadoop ecosystem applications.
emr-s3-dist-cp	2.10.0	Distributed copy application optimized for Amazon S3.
emr-s3-select	1.2.0	EMR S3Select Connector
emrfs	2.29.0	Amazon S3 connector for Hadoop ecosystem applications.
flink-client	1.6.2	Apache Flink command line client scripts and applications.
ganglia-monitor	3.7.2	Embedded Ganglia agent for Hadoop ecosystem applications along with the Ganglia monitoring agent.
ganglia-metadata-collector	3.7.2	Ganglia metadata collector for aggregating metrics from Ganglia monitoring agents.
ganglia-web	3.7.1	Web application for viewing metrics collected by the Ganglia metadata collector.
hadoop-client	2.8.5-amzn-1	Hadoop command-line clients such as 'hdfs', 'hadoop', or 'yarn'.
hadoop-hdfs-datanode	2.8.5-amzn-1	HDFS node-level service for storing blocks.
hadoop-hdfs-library	2.8.5-amzn-1	HDFS command-line client and library
hadoop-hdfs-namenode	2.8.5-amzn-1	HDFS service for tracking file names and block locations.
hadoop-httpfs-server	2.8.5-amzn-1	HTTP endpoint for HDFS operations.
hadoop-kms-server	2.8.5-amzn-1	Cryptographic key management server based on Hadoop's KeyProvider API.
hadoop-mapred	2.8.5-amzn-1	MapReduce execution engine libraries for running a MapReduce application.
hadoop-yarn-nodemanager	2.8.5-amzn-1	YARN service for managing containers on an individual node.
hadoop-yarn-resourcemanager	2.8.5-amzn-1	YARN service for allocating and managing cluster resources and distributed applications.
hadoop-yarn-timeline-server	2.8.5-amzn-1	Service for retrieving current and historical information for YARN applications.
hbase-hmaster	1.4.8	Service for an HBase cluster responsible for coordination of Regions and execution of administrative commands.
hbase-region-server	1.4.8	Service for serving one or more HBase regions.
hbase-client	1.4.8	HBase command-line client.
hbase-rest-server	1.4.8	Service providing a RESTful HTTP endpoint for HBase.
hbase-thrift-server	1.4.8	Service providing a Thrift endpoint to HBase.
hcatalog-client	2.3.4-amzn-0	The 'hcat' command line client for manipulating hcatalog-server.
hcatalog-server	2.3.4-amzn-0	Service providing HCatalog, a table and storage management layer for distributed applications.
hcatalog-webhcat-server	2.3.4-amzn-0	HTTP endpoint providing a REST interface to HCatalog.
hive-client	2.3.4-amzn-0	Hive command line client.
hive-hbase	2.3.4-amzn-0	Hive-hbase client.
hive-metastore-server	2.3.4-amzn-0	Service for accessing the Hive metastore, a semantic repository storing metadata for SQL on Hadoop operations.
hive-server2	2.3.4-amzn-0	Service for accepting Hive queries as web requests.
hue-server	4.3.0	Web application for analyzing data using Hadoop ecosystem applications
jupyterhub	0.9.4	Multi-user server for Jupyter notebooks
livy-server	0.5.0-incubating	REST interface for interacting with Apache Spark
nginx	1.12.1	nginx [engine x] is an HTTP and reverse proxy server
mahout-client	0.13.0	Library for machine learning.
mxnet	1.3.1	A flexible, scalable, and efficient library for deep learning.
mysql-server	5.5.54+	MySQL database server.
nvidia-cuda	9.2.88	Nvidia drivers and Cuda toolkit
oozie-client	5.0.0	Oozie command-line client.
oozie-server	5.0.0	Service for accepting Oozie workflow requests.
opencv	3.4.0	Open Source Computer Vision Library.
phoenix-library	4.14.0-HBase-1.4	The phoenix libraries for server and client
phoenix-query-server	4.14.0-HBase-1.4	A light weight server providing JDBC access as well as Protocol Buffers and JSON format access to the Avatica API
presto-coordinator	0.214	Service for accepting queries and managing query execution among presto-workers.
presto-worker	0.214	Service for executing pieces of a query.
pig-client	0.17.0	Pig command-line client.
r	3.4.1	The R Project for Statistical Computing
spark-client	2.4.0	Spark command-line clients.
spark-history-server	2.4.0	Web UI for viewing logged events for the lifetime of a completed Spark application.
spark-on-yarn	2.4.0	In-memory execution engine for YARN.
spark-yarn-slave	2.4.0	Apache Spark libraries needed by YARN slaves.
sqoop-client	1.4.7	Apache Sqoop command-line client.
tensorflow	1.12.0	TensorFlow open source software library for high performance numerical computation.
tez-on-yarn	0.9.1	The tez YARN application and libraries.
webserver	2.4.25+	Apache HTTP server.
zeppelin-server	0.8.0	Web-based notebook that enables interactive data analytics.
zookeeper-server	3.4.13	Centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
zookeeper-client	3.4.13	ZooKeeper command line client.

5.20.0 Configuration Classifications

emr-5.20.0 Classifications

Classifications	Description
capacity-scheduler	Change values in Hadoop's capacity-scheduler.xml file.
container-log4j	Change values in Hadoop YARN's container-log4j.properties file.
core-site	Change values in Hadoop's core-site.xml file.
emrfs-site	Change EMRFS settings.
flink-conf	Change flink-conf.yaml settings.
flink-log4j	Change Flink log4j.properties settings.
flink-log4j-yarn-session	Change Flink log4j-yarn-session.properties settings.
flink-log4j-cli	Change Flink log4j-cli.properties settings.
hadoop-env	Change values in the Hadoop environment for all Hadoop components.
hadoop-log4j	Change values in Hadoop's log4j.properties file.
hadoop-ssl-server	Change hadoop ssl server configuration
hadoop-ssl-client	Change hadoop ssl client configuration
hbase	Amazon EMR-curated settings for Apache HBase.
hbase-env	Change values in HBase's environment.
hbase-log4j	Change values in HBase's hbase-log4j.properties file.
hbase-metrics	Change values in HBase's hadoop-metrics2-hbase.properties file.
hbase-policy	Change values in HBase's hbase-policy.xml file.
hbase-site	Change values in HBase's hbase-site.xml file.
hdfs-encryption-zones	Configure HDFS encryption zones.
hdfs-site	Change values in HDFS's hdfs-site.xml.
hcatalog-env	Change values in HCatalog's environment.
hcatalog-server-jndi	Change values in HCatalog's jndi.properties.
hcatalog-server-proto-hive-site	Change values in HCatalog's proto-hive-site.xml.
hcatalog-webhcat-env	Change values in HCatalog WebHCat's environment.
hcatalog-webhcat-log4j2	Change values in HCatalog WebHCat's log4j2.properties.
hcatalog-webhcat-site	Change values in HCatalog WebHCat's webhcat-site.xml file.
hive-beeline-log4j2	Change values in Hive's beeline-log4j2.properties file.
hive-parquet-logging	Change values in Hive's parquet-logging.properties file.
hive-env	Change values in the Hive environment.
hive-exec-log4j2	Change values in Hive's hive-exec-log4j2.properties file.
hive-llap-daemon-log4j2	Change values in Hive's llap-daemon-log4j2.properties file.
hive-log4j2	Change values in Hive's hive-log4j2.properties file.
hive-site	Change values in Hive's hive-site.xml file
hiveserver2-site	Change values in Hive Server2's hiveserver2-site.xml file
hue-ini	Change values in Hue's ini file
httpfs-env	Change values in the HTTPFS environment.
httpfs-site	Change values in Hadoop's httpfs-site.xml file.
hadoop-kms-acls	Change values in Hadoop's kms-acls.xml file.
hadoop-kms-env	Change values in the Hadoop KMS environment.
hadoop-kms-log4j	Change values in Hadoop's kms-log4j.properties file.
hadoop-kms-site	Change values in Hadoop's kms-site.xml file.
jupyter-notebook-conf	Change values in Jupyter Notebook's jupyter_notebook_config.py file.
jupyter-hub-conf	Change values in JupyterHubs's jupyterhub_config.py file.
jupyter-s3-conf	Configure Jupyter Notebook S3 persistence.
jupyter-sparkmagic-conf	Change values in Sparkmagic's config.json file.
livy-conf	Change values in Livy's livy.conf file.
livy-env	Change values in the Livy environment.
livy-log4j	Change Livy log4j.properties settings.
mapred-env	Change values in the MapReduce application's environment.
mapred-site	Change values in the MapReduce application's mapred-site.xml file.
oozie-env	Change values in Oozie's environment.
oozie-log4j	Change values in Oozie's oozie-log4j.properties file.
oozie-site	Change values in Oozie's oozie-site.xml file.
phoenix-hbase-metrics	Change values in Phoenix's hadoop-metrics2-hbase.properties file.
phoenix-hbase-site	Change values in Phoenix's hbase-site.xml file.
phoenix-log4j	Change values in Phoenix's log4j.properties file.
phoenix-metrics	Change values in Phoenix's hadoop-metrics2-phoenix.properties file.
pig-env	Change values in the Pig environment.
pig-properties	Change values in Pig's pig.properties file.
pig-log4j	Change values in Pig's log4j.properties file.
presto-log	Change values in Presto's log.properties file.
presto-config	Change values in Presto's config.properties file.
presto-password-authenticator	Change values in Presto's password-authenticator.properties file.
presto-env	Change values in Presto's presto-env.sh file.
presto-node	Change values in Presto's node.properties file.
presto-connector-blackhole	Change values in Presto's blackhole.properties file.
presto-connector-cassandra	Change values in Presto's cassandra.properties file.
presto-connector-hive	Change values in Presto's hive.properties file.
presto-connector-jmx	Change values in Presto's jmx.properties file.
presto-connector-kafka	Change values in Presto's kafka.properties file.
presto-connector-localfile	Change values in Presto's localfile.properties file.
presto-connector-memory	Change values in Presto's memory.properties file.
presto-connector-mongodb	Change values in Presto's mongodb.properties file.
presto-connector-mysql	Change values in Presto's mysql.properties file.
presto-connector-postgresql	Change values in Presto's postgresql.properties file.
presto-connector-raptor	Change values in Presto's raptor.properties file.
presto-connector-redis	Change values in Presto's redis.properties file.
presto-connector-redshift	Change values in Presto's redshift.properties file.
presto-connector-tpch	Change values in Presto's tpch.properties file.
presto-connector-tpcds	Change values in Presto's tpcds.properties file.
spark	Amazon EMR-curated settings for Apache Spark.
spark-defaults	Change values in Spark's spark-defaults.conf file.
spark-env	Change values in the Spark environment.
spark-hive-site	Change values in Spark's hive-site.xml file
spark-log4j	Change values in Spark's log4j.properties file.
spark-metrics	Change values in Spark's metrics.properties file.
sqoop-env	Change values in Sqoop's environment.
sqoop-oraoop-site	Change values in Sqoop OraOop's oraoop-site.xml file.
sqoop-site	Change values in Sqoop's sqoop-site.xml file.
tez-site	Change values in Tez's tez-site.xml file.
yarn-env	Change values in the YARN environment.
yarn-site	Change values in YARN's yarn-site.xml file.
zeppelin-env	Change values in the Zeppelin environment.
zookeeper-config	Change values in ZooKeeper's zoo.cfg file.
zookeeper-log4j	Change values in ZooKeeper's log4j.properties file.

5.19.0

Amazon EMR Release 5.19.0

5.19.0 Application Versions

The diagram below depicts the application versions available in this release of Amazon EMR and the application versions in the preceding four Amazon EMR releases.

For a comprehensive history of application versions for each release of Amazon EMR, see the following diagrams:

5.19.0 Release Notes

The following release notes include information for Amazon EMR release version 5.19.0. Changes are relative to 5.18.0.

Initial release date: November 7, 2018

Last updated date: November 19, 2018

Upgrades

Hadoop 2.8.5
Flink 1.6.1
JupyterHub 0.9.4
MXNet 1.3.0
Presto 0.212
TensorFlow 1.11.0
Zookeeper 3.4.13
AWS SDK for Java 1.11.433

New Features

(Nov. 19, 2018) EMR Notebooks is a managed environment based on Jupyter Notebook. It supports Spark magic kernels for PySpark, Spark SQL, Spark R, and Scala. EMR Notebooks can be used with clusters created using Amazon EMR release version 5.18.0 and later. For more information, see Using EMR Notebooks in the Amazon EMR Management Guide.
The EMRFS S3-optimized committer is available when writing Parquet files using Spark and EMRFS. This committer improves write performance. For more information, see Using the EMRFS S3-optimized Committer.

Changes, Enhancements, and Resolved Issues

YARN
- Modified the logic that limits the application master process to running on core nodes. This functionality now uses the YARN node labels feature and properties in the yarn-site and capacity-scheduler configuration classifications. For information, see https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-instances-guidelines.html#emr-plan-spot-YARN.
Default Amazon Linux AMI for Amazon EMR
- ruby18, php56, and gcc48 are no longer installed by default. These can be installed if desired using yum.
- The aws-java-sdk ruby gem is no longer installed by default. It can be installed using gem install aws-java-sdk, if desired. Specific components can also be installed. For example, gem install aws-java-sdk-s3.

Known Issues

EMR Notebooks—In some circumstances, with multiple notebook editors open, the notebook editor may appear unable to connect to the cluster. If this happens, clear browser cookies and then reopen notebook editors.
CloudWatch ContainerPending Metric and Automatic Scaling—(Fixed in 5.20.0)Amazon EMR may emit a negative value for ContainerPending. If ContainerPending is used in an automatic scaling rule, automatic scaling does not behave as expected. Avoid using ContainerPending with automatic scaling.
In Amazon EMR version 5.19.0, 5.20.0, and 5.21.0, YARN node labels are stored in an HDFS directory. In some situations, this leads to core node startup delays and then causes cluster time-out and launch failure. Beginning with Amazon EMR 5.22.0, this issue is resolved. YARN node labels are stored on the local disk of each cluster node, avoiding dependencies on HDFS.

5.19.0 Component Versions

Component	Version	Description
aws-sagemaker-spark-sdk	1.2.0	Amazon SageMaker Spark SDK
emr-ddb	4.7.0	Amazon DynamoDB connector for Hadoop ecosystem applications.
emr-goodies	2.5.1	Extra convenience libraries for the Hadoop ecosystem.
emr-kinesis	3.4.0	Amazon Kinesis connector for Hadoop ecosystem applications.
emr-s3-dist-cp	2.10.0	Distributed copy application optimized for Amazon S3.
emr-s3-select	1.1.0	EMR S3Select Connector
emrfs	2.28.0	Amazon S3 connector for Hadoop ecosystem applications.
flink-client	1.6.1	Apache Flink command line client scripts and applications.
ganglia-monitor	3.7.2	Embedded Ganglia agent for Hadoop ecosystem applications along with the Ganglia monitoring agent.
ganglia-metadata-collector	3.7.2	Ganglia metadata collector for aggregating metrics from Ganglia monitoring agents.
ganglia-web	3.7.1	Web application for viewing metrics collected by the Ganglia metadata collector.
hadoop-client	2.8.5-amzn-0	Hadoop command-line clients such as 'hdfs', 'hadoop', or 'yarn'.
hadoop-hdfs-datanode	2.8.5-amzn-0	HDFS node-level service for storing blocks.
hadoop-hdfs-library	2.8.5-amzn-0	HDFS command-line client and library
hadoop-hdfs-namenode	2.8.5-amzn-0	HDFS service for tracking file names and block locations.
hadoop-httpfs-server	2.8.5-amzn-0	HTTP endpoint for HDFS operations.
hadoop-kms-server	2.8.5-amzn-0	Cryptographic key management server based on Hadoop's KeyProvider API.
hadoop-mapred	2.8.5-amzn-0	MapReduce execution engine libraries for running a MapReduce application.
hadoop-yarn-nodemanager	2.8.5-amzn-0	YARN service for managing containers on an individual node.
hadoop-yarn-resourcemanager	2.8.5-amzn-0	YARN service for allocating and managing cluster resources and distributed applications.
hadoop-yarn-timeline-server	2.8.5-amzn-0	Service for retrieving current and historical information for YARN applications.
hbase-hmaster	1.4.7	Service for an HBase cluster responsible for coordination of Regions and execution of administrative commands.
hbase-region-server	1.4.7	Service for serving one or more HBase regions.
hbase-client	1.4.7	HBase command-line client.
hbase-rest-server	1.4.7	Service providing a RESTful HTTP endpoint for HBase.
hbase-thrift-server	1.4.7	Service providing a Thrift endpoint to HBase.
hcatalog-client	2.3.3-amzn-2	The 'hcat' command line client for manipulating hcatalog-server.
hcatalog-server	2.3.3-amzn-2	Service providing HCatalog, a table and storage management layer for distributed applications.
hcatalog-webhcat-server	2.3.3-amzn-2	HTTP endpoint providing a REST interface to HCatalog.
hive-client	2.3.3-amzn-2	Hive command line client.
hive-hbase	2.3.3-amzn-2	Hive-hbase client.
hive-metastore-server	2.3.3-amzn-2	Service for accessing the Hive metastore, a semantic repository storing metadata for SQL on Hadoop operations.
hive-server2	2.3.3-amzn-2	Service for accepting Hive queries as web requests.
hue-server	4.2.0	Web application for analyzing data using Hadoop ecosystem applications
jupyterhub	0.9.4	Multi-user server for Jupyter notebooks
livy-server	0.5.0-incubating	REST interface for interacting with Apache Spark
nginx	1.12.1	nginx [engine x] is an HTTP and reverse proxy server
mahout-client	0.13.0	Library for machine learning.
mxnet	1.3.0	A flexible, scalable, and efficient library for deep learning.
mysql-server	5.5.54+	MySQL database server.
nvidia-cuda	9.2.88	Nvidia drivers and Cuda toolkit
oozie-client	5.0.0	Oozie command-line client.
oozie-server	5.0.0	Service for accepting Oozie workflow requests.
opencv	3.4.0	Open Source Computer Vision Library.
phoenix-library	4.14.0-HBase-1.4	The phoenix libraries for server and client
phoenix-query-server	4.14.0-HBase-1.4	A light weight server providing JDBC access as well as Protocol Buffers and JSON format access to the Avatica API
presto-coordinator	0.212	Service for accepting queries and managing query execution among presto-workers.
presto-worker	0.212	Service for executing pieces of a query.
pig-client	0.17.0	Pig command-line client.
r	3.4.1	The R Project for Statistical Computing
spark-client	2.3.2	Spark command-line clients.
spark-history-server	2.3.2	Web UI for viewing logged events for the lifetime of a completed Spark application.
spark-on-yarn	2.3.2	In-memory execution engine for YARN.
spark-yarn-slave	2.3.2	Apache Spark libraries needed by YARN slaves.
sqoop-client	1.4.7	Apache Sqoop command-line client.
tensorflow	1.11.0	TensorFlow open source software library for high performance numerical computation.
tez-on-yarn	0.8.4	The tez YARN application and libraries.
webserver	2.4.25+	Apache HTTP server.
zeppelin-server	0.8.0	Web-based notebook that enables interactive data analytics.
zookeeper-server	3.4.13	Centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
zookeeper-client	3.4.13	ZooKeeper command line client.

5.19.0 Configuration Classifications

emr-5.19.0 Classifications

Classifications	Description
capacity-scheduler	Change values in Hadoop's capacity-scheduler.xml file.
container-log4j	Change values in Hadoop YARN's container-log4j.properties file.
core-site	Change values in Hadoop's core-site.xml file.
emrfs-site	Change EMRFS settings.
flink-conf	Change flink-conf.yaml settings.
flink-log4j	Change Flink log4j.properties settings.
flink-log4j-yarn-session	Change Flink log4j-yarn-session.properties settings.
flink-log4j-cli	Change Flink log4j-cli.properties settings.
hadoop-env	Change values in the Hadoop environment for all Hadoop components.
hadoop-log4j	Change values in Hadoop's log4j.properties file.
hadoop-ssl-server	Change hadoop ssl server configuration
hadoop-ssl-client	Change hadoop ssl client configuration
hbase	Amazon EMR-curated settings for Apache HBase.
hbase-env	Change values in HBase's environment.
hbase-log4j	Change values in HBase's hbase-log4j.properties file.
hbase-metrics	Change values in HBase's hadoop-metrics2-hbase.properties file.
hbase-policy	Change values in HBase's hbase-policy.xml file.
hbase-site	Change values in HBase's hbase-site.xml file.
hdfs-encryption-zones	Configure HDFS encryption zones.
hdfs-site	Change values in HDFS's hdfs-site.xml.
hcatalog-env	Change values in HCatalog's environment.
hcatalog-server-jndi	Change values in HCatalog's jndi.properties.
hcatalog-server-proto-hive-site	Change values in HCatalog's proto-hive-site.xml.
hcatalog-webhcat-env	Change values in HCatalog WebHCat's environment.
hcatalog-webhcat-log4j2	Change values in HCatalog WebHCat's log4j2.properties.
hcatalog-webhcat-site	Change values in HCatalog WebHCat's webhcat-site.xml file.
hive-beeline-log4j2	Change values in Hive's beeline-log4j2.properties file.
hive-parquet-logging	Change values in Hive's parquet-logging.properties file.
hive-env	Change values in the Hive environment.
hive-exec-log4j2	Change values in Hive's hive-exec-log4j2.properties file.
hive-llap-daemon-log4j2	Change values in Hive's llap-daemon-log4j2.properties file.
hive-log4j2	Change values in Hive's hive-log4j2.properties file.
hive-site	Change values in Hive's hive-site.xml file
hiveserver2-site	Change values in Hive Server2's hiveserver2-site.xml file
hue-ini	Change values in Hue's ini file
httpfs-env	Change values in the HTTPFS environment.
httpfs-site	Change values in Hadoop's httpfs-site.xml file.
hadoop-kms-acls	Change values in Hadoop's kms-acls.xml file.
hadoop-kms-env	Change values in the Hadoop KMS environment.
hadoop-kms-log4j	Change values in Hadoop's kms-log4j.properties file.
hadoop-kms-site	Change values in Hadoop's kms-site.xml file.
jupyter-notebook-conf	Change values in Jupyter Notebook's jupyter_notebook_config.py file.
jupyter-hub-conf	Change values in JupyterHubs's jupyterhub_config.py file.
jupyter-s3-conf	Configure Jupyter Notebook S3 persistence.
jupyter-sparkmagic-conf	Change values in Sparkmagic's config.json file.
livy-conf	Change values in Livy's livy.conf file.
livy-env	Change values in the Livy environment.
livy-log4j	Change Livy log4j.properties settings.
mapred-env	Change values in the MapReduce application's environment.
mapred-site	Change values in the MapReduce application's mapred-site.xml file.
oozie-env	Change values in Oozie's environment.
oozie-log4j	Change values in Oozie's oozie-log4j.properties file.
oozie-site	Change values in Oozie's oozie-site.xml file.
phoenix-hbase-metrics	Change values in Phoenix's hadoop-metrics2-hbase.properties file.
phoenix-hbase-site	Change values in Phoenix's hbase-site.xml file.
phoenix-log4j	Change values in Phoenix's log4j.properties file.
phoenix-metrics	Change values in Phoenix's hadoop-metrics2-phoenix.properties file.
pig-env	Change values in the Pig environment.
pig-properties	Change values in Pig's pig.properties file.
pig-log4j	Change values in Pig's log4j.properties file.
presto-log	Change values in Presto's log.properties file.
presto-config	Change values in Presto's config.properties file.
presto-password-authenticator	Change values in Presto's password-authenticator.properties file.
presto-env	Change values in Presto's presto-env.sh file.
presto-node	Change values in Presto's node.properties file.
presto-connector-blackhole	Change values in Presto's blackhole.properties file.
presto-connector-cassandra	Change values in Presto's cassandra.properties file.
presto-connector-hive	Change values in Presto's hive.properties file.
presto-connector-jmx	Change values in Presto's jmx.properties file.
presto-connector-kafka	Change values in Presto's kafka.properties file.
presto-connector-localfile	Change values in Presto's localfile.properties file.
presto-connector-memory	Change values in Presto's memory.properties file.
presto-connector-mongodb	Change values in Presto's mongodb.properties file.
presto-connector-mysql	Change values in Presto's mysql.properties file.
presto-connector-postgresql	Change values in Presto's postgresql.properties file.
presto-connector-raptor	Change values in Presto's raptor.properties file.
presto-connector-redis	Change values in Presto's redis.properties file.
presto-connector-redshift	Change values in Presto's redshift.properties file.
presto-connector-tpch	Change values in Presto's tpch.properties file.
presto-connector-tpcds	Change values in Presto's tpcds.properties file.
spark	Amazon EMR-curated settings for Apache Spark.
spark-defaults	Change values in Spark's spark-defaults.conf file.
spark-env	Change values in the Spark environment.
spark-hive-site	Change values in Spark's hive-site.xml file
spark-log4j	Change values in Spark's log4j.properties file.
spark-metrics	Change values in Spark's metrics.properties file.
sqoop-env	Change values in Sqoop's environment.
sqoop-oraoop-site	Change values in Sqoop OraOop's oraoop-site.xml file.
sqoop-site	Change values in Sqoop's sqoop-site.xml file.
tez-site	Change values in Tez's tez-site.xml file.
yarn-env	Change values in the YARN environment.
yarn-site	Change values in YARN's yarn-site.xml file.
zeppelin-env	Change values in the Zeppelin environment.
zookeeper-config	Change values in ZooKeeper's zoo.cfg file.
zookeeper-log4j	Change values in ZooKeeper's log4j.properties file.

5.18.0

Amazon EMR Release 5.18.0

5.18.0 Application Versions

The diagram below depicts the application versions available in this release of Amazon EMR and the application versions in the preceding four Amazon EMR releases.

For a comprehensive history of application versions for each release of Amazon EMR, see the following diagrams:

5.18.0 Release Notes

The following release notes include information for Amazon EMR release version 5.18.0. Changes are relative to 5.17.0.

Initial release date: October 24, 2018

Upgrades

Flink 1.6.0
HBase 1.4.7
Presto 0.210
Spark 2.3.2
Zeppelin 0.8.0

New Features

Beginning with Amazon EMR 5.18.0, you can use the Amazon EMR artifact repository to build your job code against the exact versions of libraries and dependencies that are available with specific Amazon EMR release versions. For more information, see Checking Dependencies Using the Amazon EMR Artifact Repository.

Changes, Enhancements, and Resolved Issues

Hive
- Added support for S3 Select. For more information, see Using S3 Select with Hive to Improve Performance.
Presto
- Added support for S3 Select Pushdown. For more information, see Using S3 Select Pushdown with Presto to Improve Performance.
Spark
- The default log4j configuration for Spark has been changed to roll container logs hourly for Spark streaming jobs. This helps prevent the deletion of logs for long-running Spark streaming jobs.

5.18.0 Component Versions

Component	Version	Description
aws-sagemaker-spark-sdk	1.1.3	Amazon SageMaker Spark SDK
emr-ddb	4.6.0	Amazon DynamoDB connector for Hadoop ecosystem applications.
emr-goodies	2.5.0	Extra convenience libraries for the Hadoop ecosystem.
emr-kinesis	3.4.0	Amazon Kinesis connector for Hadoop ecosystem applications.
emr-s3-dist-cp	2.10.0	Distributed copy application optimized for Amazon S3.
emr-s3-select	1.1.0	EMR S3Select Connector
emrfs	2.27.0	Amazon S3 connector for Hadoop ecosystem applications.
flink-client	1.6.0	Apache Flink command line client scripts and applications.
ganglia-monitor	3.7.2	Embedded Ganglia agent for Hadoop ecosystem applications along with the Ganglia monitoring agent.
ganglia-metadata-collector	3.7.2	Ganglia metadata collector for aggregating metrics from Ganglia monitoring agents.
ganglia-web	3.7.1	Web application for viewing metrics collected by the Ganglia metadata collector.
hadoop-client	2.8.4-amzn-1	Hadoop command-line clients such as 'hdfs', 'hadoop', or 'yarn'.
hadoop-hdfs-datanode	2.8.4-amzn-1	HDFS node-level service for storing blocks.
hadoop-hdfs-library	2.8.4-amzn-1	HDFS command-line client and library
hadoop-hdfs-namenode	2.8.4-amzn-1	HDFS service for tracking file names and block locations.
hadoop-httpfs-server	2.8.4-amzn-1	HTTP endpoint for HDFS operations.
hadoop-kms-server	2.8.4-amzn-1	Cryptographic key management server based on Hadoop's KeyProvider API.
hadoop-mapred	2.8.4-amzn-1	MapReduce execution engine libraries for running a MapReduce application.
hadoop-yarn-nodemanager	2.8.4-amzn-1	YARN service for managing containers on an individual node.
hadoop-yarn-resourcemanager	2.8.4-amzn-1	YARN service for allocating and managing cluster resources and distributed applications.
hadoop-yarn-timeline-server	2.8.4-amzn-1	Service for retrieving current and historical information for YARN applications.
hbase-hmaster	1.4.7	Service for an HBase cluster responsible for coordination of Regions and execution of administrative commands.
hbase-region-server	1.4.7	Service for serving one or more HBase regions.
hbase-client	1.4.7	HBase command-line client.
hbase-rest-server	1.4.7	Service providing a RESTful HTTP endpoint for HBase.
hbase-thrift-server	1.4.7	Service providing a Thrift endpoint to HBase.
hcatalog-client	2.3.3-amzn-2	The 'hcat' command line client for manipulating hcatalog-server.
hcatalog-server	2.3.3-amzn-2	Service providing HCatalog, a table and storage management layer for distributed applications.
hcatalog-webhcat-server	2.3.3-amzn-2	HTTP endpoint providing a REST interface to HCatalog.
hive-client	2.3.3-amzn-2	Hive command line client.
hive-hbase	2.3.3-amzn-2	Hive-hbase client.
hive-metastore-server	2.3.3-amzn-2	Service for accessing the Hive metastore, a semantic repository storing metadata for SQL on Hadoop operations.
hive-server2	2.3.3-amzn-2	Service for accepting Hive queries as web requests.
hue-server	4.2.0	Web application for analyzing data using Hadoop ecosystem applications
jupyterhub	0.8.1	Multi-user server for Jupyter notebooks
livy-server	0.5.0-incubating	REST interface for interacting with Apache Spark
nginx	1.12.1	nginx [engine x] is an HTTP and reverse proxy server
mahout-client	0.13.0	Library for machine learning.
mxnet	1.2.0	A flexible, scalable, and efficient library for deep learning.
mysql-server	5.5.54+	MySQL database server.
nvidia-cuda	9.2.88	Nvidia drivers and Cuda toolkit
oozie-client	5.0.0	Oozie command-line client.
oozie-server	5.0.0	Service for accepting Oozie workflow requests.
opencv	3.4.0	Open Source Computer Vision Library.
phoenix-library	4.14.0-HBase-1.4	The phoenix libraries for server and client
phoenix-query-server	4.14.0-HBase-1.4	A light weight server providing JDBC access as well as Protocol Buffers and JSON format access to the Avatica API
presto-coordinator	0.210	Service for accepting queries and managing query execution among presto-workers.
presto-worker	0.210	Service for executing pieces of a query.
pig-client	0.17.0	Pig command-line client.
r	3.4.1	The R Project for Statistical Computing
spark-client	2.3.2	Spark command-line clients.
spark-history-server	2.3.2	Web UI for viewing logged events for the lifetime of a completed Spark application.
spark-on-yarn	2.3.2	In-memory execution engine for YARN.
spark-yarn-slave	2.3.2	Apache Spark libraries needed by YARN slaves.
sqoop-client	1.4.7	Apache Sqoop command-line client.
tensorflow	1.9.0	TensorFlow open source software library for high performance numerical computation.
tez-on-yarn	0.8.4	The tez YARN application and libraries.
webserver	2.4.25+	Apache HTTP server.
zeppelin-server	0.8.0	Web-based notebook that enables interactive data analytics.
zookeeper-server	3.4.12	Centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
zookeeper-client	3.4.12	ZooKeeper command line client.

5.18.0 Configuration Classifications

emr-5.18.0 Classifications

Classifications	Description
capacity-scheduler	Change values in Hadoop's capacity-scheduler.xml file.
container-log4j	Change values in Hadoop YARN's container-log4j.properties file.
core-site	Change values in Hadoop's core-site.xml file.
emrfs-site	Change EMRFS settings.
flink-conf	Change flink-conf.yaml settings.
flink-log4j	Change Flink log4j.properties settings.
flink-log4j-yarn-session	Change Flink log4j-yarn-session.properties settings.
flink-log4j-cli	Change Flink log4j-cli.properties settings.
hadoop-env	Change values in the Hadoop environment for all Hadoop components.
hadoop-log4j	Change values in Hadoop's log4j.properties file.
hadoop-ssl-server	Change hadoop ssl server configuration
hadoop-ssl-client	Change hadoop ssl client configuration
hbase	Amazon EMR-curated settings for Apache HBase.
hbase-env	Change values in HBase's environment.
hbase-log4j	Change values in HBase's hbase-log4j.properties file.
hbase-metrics	Change values in HBase's hadoop-metrics2-hbase.properties file.
hbase-policy	Change values in HBase's hbase-policy.xml file.
hbase-site	Change values in HBase's hbase-site.xml file.
hdfs-encryption-zones	Configure HDFS encryption zones.
hdfs-site	Change values in HDFS's hdfs-site.xml.
hcatalog-env	Change values in HCatalog's environment.
hcatalog-server-jndi	Change values in HCatalog's jndi.properties.
hcatalog-server-proto-hive-site	Change values in HCatalog's proto-hive-site.xml.
hcatalog-webhcat-env	Change values in HCatalog WebHCat's environment.
hcatalog-webhcat-log4j2	Change values in HCatalog WebHCat's log4j2.properties.
hcatalog-webhcat-site	Change values in HCatalog WebHCat's webhcat-site.xml file.
hive-beeline-log4j2	Change values in Hive's beeline-log4j2.properties file.
hive-parquet-logging	Change values in Hive's parquet-logging.properties file.
hive-env	Change values in the Hive environment.
hive-exec-log4j2	Change values in Hive's hive-exec-log4j2.properties file.
hive-llap-daemon-log4j2	Change values in Hive's llap-daemon-log4j2.properties file.
hive-log4j2	Change values in Hive's hive-log4j2.properties file.
hive-site	Change values in Hive's hive-site.xml file
hiveserver2-site	Change values in Hive Server2's hiveserver2-site.xml file
hue-ini	Change values in Hue's ini file
httpfs-env	Change values in the HTTPFS environment.
httpfs-site	Change values in Hadoop's httpfs-site.xml file.
hadoop-kms-acls	Change values in Hadoop's kms-acls.xml file.
hadoop-kms-env	Change values in the Hadoop KMS environment.
hadoop-kms-log4j	Change values in Hadoop's kms-log4j.properties file.
hadoop-kms-site	Change values in Hadoop's kms-site.xml file.
jupyter-notebook-conf	Change values in Jupyter Notebook's jupyter_notebook_config.py file.
jupyter-hub-conf	Change values in JupyterHubs's jupyterhub_config.py file.
jupyter-s3-conf	Configure Jupyter Notebook S3 persistence.
jupyter-sparkmagic-conf	Change values in Sparkmagic's config.json file.
livy-conf	Change values in Livy's livy.conf file.
livy-env	Change values in the Livy environment.
livy-log4j	Change Livy log4j.properties settings.
mapred-env	Change values in the MapReduce application's environment.
mapred-site	Change values in the MapReduce application's mapred-site.xml file.
oozie-env	Change values in Oozie's environment.
oozie-log4j	Change values in Oozie's oozie-log4j.properties file.
oozie-site	Change values in Oozie's oozie-site.xml file.
phoenix-hbase-metrics	Change values in Phoenix's hadoop-metrics2-hbase.properties file.
phoenix-hbase-site	Change values in Phoenix's hbase-site.xml file.
phoenix-log4j	Change values in Phoenix's log4j.properties file.
phoenix-metrics	Change values in Phoenix's hadoop-metrics2-phoenix.properties file.
pig-env	Change values in the Pig environment.
pig-properties	Change values in Pig's pig.properties file.
pig-log4j	Change values in Pig's log4j.properties file.
presto-log	Change values in Presto's log.properties file.
presto-config	Change values in Presto's config.properties file.
presto-password-authenticator	Change values in Presto's password-authenticator.properties file.
presto-env	Change values in Presto's presto-env.sh file.
presto-node	Change values in Presto's node.properties file.
presto-connector-blackhole	Change values in Presto's blackhole.properties file.
presto-connector-cassandra	Change values in Presto's cassandra.properties file.
presto-connector-hive	Change values in Presto's hive.properties file.
presto-connector-jmx	Change values in Presto's jmx.properties file.
presto-connector-kafka	Change values in Presto's kafka.properties file.
presto-connector-localfile	Change values in Presto's localfile.properties file.
presto-connector-mongodb	Change values in Presto's mongodb.properties file.
presto-connector-mysql	Change values in Presto's mysql.properties file.
presto-connector-postgresql	Change values in Presto's postgresql.properties file.
presto-connector-raptor	Change values in Presto's raptor.properties file.
presto-connector-redis	Change values in Presto's redis.properties file.
presto-connector-redshift	Change values in Presto's redshift.properties file.
presto-connector-tpch	Change values in Presto's tpch.properties file.
spark	Amazon EMR-curated settings for Apache Spark.
spark-defaults	Change values in Spark's spark-defaults.conf file.
spark-env	Change values in the Spark environment.
spark-hive-site	Change values in Spark's hive-site.xml file
spark-log4j	Change values in Spark's log4j.properties file.
spark-metrics	Change values in Spark's metrics.properties file.
sqoop-env	Change values in Sqoop's environment.
sqoop-oraoop-site	Change values in Sqoop OraOop's oraoop-site.xml file.
sqoop-site	Change values in Sqoop's sqoop-site.xml file.
tez-site	Change values in Tez's tez-site.xml file.
yarn-env	Change values in the YARN environment.
yarn-site	Change values in YARN's yarn-site.xml file.
zeppelin-env	Change values in the Zeppelin environment.
zookeeper-config	Change values in ZooKeeper's zoo.cfg file.
zookeeper-log4j	Change values in ZooKeeper's log4j.properties file.

5.17.x

There are multiple releases within the 5.17 series. Choose a link below to see information for a specific release within this tab.

5.17.1 (Latest) | 5.17.0

Amazon EMR Release 5.17.1

Release 5.17.1 Application Versions

The diagram below depicts the application versions available in this release of Amazon EMR and the application versions in the preceding four Amazon EMR releases.

For a comprehensive history of application versions for each release of Amazon EMR, see the following diagrams:

Release 5.17.1 Release Notes

The following release notes include information for Amazon EMR release version 5.17.1. Changes are relative to 5.17.0.

Initial release date: July 18, 2019

Changes, Enhancements, and Resolved Issues

Updated the default Amazon Linux AMI for EMR to include important Linux kernel security updates, including the TCP SACK Denial of Service Issue (AWS-2019-005).

Release 5.17.1 Component Versions

Component	Version	Description
aws-sagemaker-spark-sdk	1.1.3	Amazon SageMaker Spark SDK
emr-ddb	4.6.0	Amazon DynamoDB connector for Hadoop ecosystem applications.
emr-goodies	2.5.0	Extra convenience libraries for the Hadoop ecosystem.
emr-kinesis	3.4.0	Amazon Kinesis connector for Hadoop ecosystem applications.
emr-s3-dist-cp	2.10.0	Distributed copy application optimized for Amazon S3.
emr-s3-select	1.0.0	EMR S3Select Connector
emrfs	2.26.0	Amazon S3 connector for Hadoop ecosystem applications.
flink-client	1.5.2	Apache Flink command line client scripts and applications.
ganglia-monitor	3.7.2	Embedded Ganglia agent for Hadoop ecosystem applications along with the Ganglia monitoring agent.
ganglia-metadata-collector	3.7.2	Ganglia metadata collector for aggregating metrics from Ganglia monitoring agents.
ganglia-web	3.7.1	Web application for viewing metrics collected by the Ganglia metadata collector.
hadoop-client	2.8.4-amzn-1	Hadoop command-line clients such as 'hdfs', 'hadoop', or 'yarn'.
hadoop-hdfs-datanode	2.8.4-amzn-1	HDFS node-level service for storing blocks.
hadoop-hdfs-library	2.8.4-amzn-1	HDFS command-line client and library
hadoop-hdfs-namenode	2.8.4-amzn-1	HDFS service for tracking file names and block locations.
hadoop-httpfs-server	2.8.4-amzn-1	HTTP endpoint for HDFS operations.
hadoop-kms-server	2.8.4-amzn-1	Cryptographic key management server based on Hadoop's KeyProvider API.
hadoop-mapred	2.8.4-amzn-1	MapReduce execution engine libraries for running a MapReduce application.
hadoop-yarn-nodemanager	2.8.4-amzn-1	YARN service for managing containers on an individual node.
hadoop-yarn-resourcemanager	2.8.4-amzn-1	YARN service for allocating and managing cluster resources and distributed applications.
hadoop-yarn-timeline-server	2.8.4-amzn-1	Service for retrieving current and historical information for YARN applications.
hbase-hmaster	1.4.6	Service for an HBase cluster responsible for coordination of Regions and execution of administrative commands.
hbase-region-server	1.4.6	Service for serving one or more HBase regions.
hbase-client	1.4.6	HBase command-line client.
hbase-rest-server	1.4.6	Service providing a RESTful HTTP endpoint for HBase.
hbase-thrift-server	1.4.6