Amazon EMR
Amazon EMR Release Guide

Using the AWS Glue Data Catalog as the Metastore for Hive

Using Amazon EMR version 5.8.0 or later, you can configure Hive to use the AWS Glue Data Catalog as its metastore. We recommend this configuration when you require a persistent metastore or a metastore shared by different clusters, services, applications, or AWS accounts.

AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it simple and cost-effective to categorize your data, clean it, enrich it, and move it reliably between various data stores. The AWS Glue Data Catalog provides a unified metadata repository across a variety of data sources and data formats, integrating with Amazon EMR as well as Amazon RDS, Amazon Redshift, Redshift Spectrum, Athena, and any application compatible with the Apache Hive metastore. AWS Glue crawlers can automatically infer schema from source data in Amazon S3 and store the associated metadata in the Data Catalog. For more information about the Data Catalog, see Populating the AWS Glue Data Catalog in the AWS Glue Developer Guide.

Separate charges apply for AWS Glue. There is a monthly rate for storing and accessing the metadata in the Data Catalog, an hourly rate billed per minute for AWS Glue ETL jobs and crawler runtime, and an hourly rate billed per minute for each provisioned development endpoint. The Data Catalog allows you to store up to a million objects at no charge. If you store more than a million objects, you are charged USD$1 for each 100,000 objects over a million. An object in the Data Catalog is a table, partition, or database. For more information, see Glue Pricing.

Important

If you created tables using Amazon Athena or Amazon Redshift Spectrum before August 14, 2017, databases and tables are stored in an Athena-managed catalog, which is separate from the AWS Glue Data Catalog. To integrate Amazon EMR with these tables, you must upgrade to the AWS Glue Data Catalog. For more information, see Upgrading to the AWS Glue Data Catalog in the Amazon Athena User Guide.

Specifying AWS Glue Data Catalog as the Metastore

You can specify the AWS Glue Data Catalog as the metastore using the AWS Management Console, AWS CLI, or Amazon EMR API. When you create a cluster using the CLI or API, you use the configuration classification for Hive to specify the Data Catalog. In addition, with Amazon EMR 5.16.0 and later, you can use the configuration classification to specify a Data Catalog in a different AWS account. When you create a cluster using the console, you can specify the Data Catalog using Advanced Options or Quick Options.

Note

The option to use the Data Catalog is also available with HCatalog because Hive is installed with HCatalog.

To specify AWS Glue Data Catalog as the metastore using the console

  1. Open the Amazon EMR console at https://console.aws.amazon.com/elasticmapreduce/.

  2. Choose Create cluster, Go to advanced options.

  3. For Release, choose emr-5.8.0 or later.

  4. Under Release, select Hive or HCatalog.

  5. Under AWS Glue Data Catalog settings select Use for Hive table metadata.

  6. Choose other options for your cluster as appropriate, choose Next, and then configure other cluster options as appropriate for your application.

To specify the AWS Glue Data Catalog as the metastore using the configuration classification

For more information about specifying a configuration classification using the AWS CLI and EMR API, see Configuring Applications.

  • Specify the value for hive.metastore.client.factory.class using the hive-site configuration classification as shown in the following example:

    [ { "Classification": "hive-site", "Properties": { "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory" } } ]

    To specify a Data Catalog in a different AWS account, add the hive.metastore.glue.catalogid property as shown in the following example. Replace acct-id with the AWS account of the Data Catalog.

    [ { "Classification": "hive-site", "Properties": { "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory", "hive.metastore.glue.catalogid": "acct-id" } } ]

IAM Permissions

The EC2 instance profile for a cluster must have IAM permissions for AWS Glue actions. In addition, if you enable encryption for AWS Glue Data Catalog objects, the role must also be allowed to encrypt, decrypt and generate the customer master key (CMK) used for encryption.

Permissions for AWS Glue Actions

The default AmazonElasticMapReduceforEC2Role managed policy attached to EMR_EC2_DefaultRole allows the required AWS Glue actions. If you use the default EC2 instance profile, no action is required. However, if you specify a custom EC2 instance profile and permissions when you create a cluster, ensure that the appropriate AWS Glue actions are allowed. Use the AmazonElasticMapReduceforEC2Role managed policy as a starting point. For a listing of AWS Glue actions, see Default Contents of AmazonElasticMapReduceforEC2Role Permissions Policy in the Amazon EMR Management Guide.

Permissions for Encrypting and Decrypting AWS Glue Data Catalog

This section is about the encryption feature of the AWS Glue Data Catalog. For more information about AWS Glue Data Catalog encryption, see Encrypting Your Data Catalog in the AWS Glue Developer Guide.

If you enable encryption for AWS Glue Data Catalog objects using AWS managed CMKs for AWS Glue, and the cluster that accesses the AWS Glue Data Catalog is within the same AWS account, you don't need to update the permissions policy attached to the EC2 instance profile. If you use a customer managed CMK, or if the cluster is in a different AWS account, you must update the permissions policy so that the EC2 instance profile has permission to encrypt and decrypt using the key. The contents of the following policy statement needs to be added regardless of whether you use the default permissions policy, AmazonElasticMapReduceforEC2Role, or you use a custom permissions policy attached to a custom EC2 instance profile.

[ { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:Encrypt", "kms:GenerateDataKey" ], "Resource": "arn:aws:kms:region:acct-id:key/12345678-1234-1234-1234-123456789012" } ] } ]

Resource-Based Permissions

If you use AWS Glue in conjunction with Hive, Spark, or Presto in Amazon EMR, AWS Glue supports resource-based policies to control access to Data Catalog resources. These resources include databases, tables, connections, and user-defined functions. For more information, see AWS Glue Resource Policies in the AWS Glue Developer Guide.

When using resource-based policies to limit access to AWS Glue from within Amazon EMR, the principal that you specify in the permissions policy must be the role ARN associated with the EC2 instance profile that is specified when a cluster is created. For example, if you use the default EC2 instance profile, this role ARN has the following format:

arn:aws:iam::acct-id:role/EMR_EC2_DefaultRole

The acct-id can be different from the AWS Glue account ID. This enables access from EMR clusters in different accounts. You can specify multiple principals, each from a different account.

Considerations When Using AWS Glue Data Catalog

Consider the following items when using the AWS Glue Data Catalog as the metastore with Hive:

  • Adding auxiliary JARs using the Hive shell is not supported. As a workaround, use the hive-site configuration classification to set the hive.aux.jars.path property, which adds auxiliary JARs into the Hive classpath.

  • The VALUES keyword is not supported.

  • Hive transactions are not supported.

  • Renaming tables from within AWS Glue is not supported.

  • When you create a Hive table without specifying a LOCATION, the table definition is stored in the location specified by the hive.metastore.warehouse.dir property. By default, this is a location in HDFS. If another cluster needs to access the table, it fails unless it has adequate permissions to the cluster that created the table. Furthermore, because HDFS storage is transient, if the cluster terminates, the table definition is lost, and the table must be recreated. We recommend that you specify a LOCATION in Amazon S3 when you create a Hive table using AWS Glue. Alternatively, you can use the hive-site configuration classification to specify a location in Amazon S3 for hive.metastore.warehouse.dir, which applies to all Hive tables. If a table is created in an HDFS location and the cluster that created it is still running, you can update the table location to Amazon S3 from within AWS Glue. For more information, see Working with Tables on the AWS Glue Console in the AWS Glue Developer Guide.

  • Partition values containing quotes and apostrophes are not supported, for example, PARTITION (owner="Doe's").

  • Column statistics are not supported.

  • Using Hive authorization is not supported. As an alternative, consider using AWS Glue Resource-Based Policies. For more information, see Use Resource-Based Policies for Amazon EMR Access to AWS Glue Data Catalog.

  • Hive constraints are not supported.

  • Cost-based Optimization in Hive is not supported. Changing the value of hive.cbo.enable to true is not supported.

  • Setting hive.metastore.partition.inherit.table.properties is not supported.

  • Using the following metastore constants is not supported: BUCKET_COUNT, BUCKET_FIELD_NAME, DDL_TIME, FIELD_TO_DIMENSION, FILE_INPUT_FORMAT, FILE_OUTPUT_FORMAT, HIVE_FILTER_FIELD_LAST_ACCESS, HIVE_FILTER_FIELD_OWNER, HIVE_FILTER_FIELD_PARAMS, IS_ARCHIVED, META_TABLE_COLUMNS, META_TABLE_COLUMN_TYPES, META_TABLE_DB, META_TABLE_LOCATION, META_TABLE_NAME, META_TABLE_PARTITION_COLUMNS, META_TABLE_SERDE, META_TABLE_STORAGE, ORIGINAL_LOCATION.

  • When you use a predicate expression, explicit values must be on the right side of the comparison operator, or queries might fail.

    • Correct: SELECT * FROM mytable WHERE time > 11

    • Incorrect: SELECT * FROM mytable WHERE 11 > time

  • We do not recommend using user-defined functions (UDFs) in predicate expressions. Queries may fail because of the way Hive tries to optimize query execution.

  • Temporary tables are not supported.

  • We recommend creating tables using applications through Amazon EMR rather than creating them directly using AWS Glue. Creating a table through AWS Glue may cause required fields to be missing and cause query exceptions.