AWS Glue Construct Library

--- ![cfn-resources: Stable](https://img.shields.io/badge/cfn--resources-stable-success.svg?style=for-the-badge) > All classes with the `Cfn` prefix in this module ([CFN Resources](https://docs.aws.amazon.com/cdk/latest/guide/constructs.html#constructs_lib)) are always stable and safe to use. ![cdk-constructs: Experimental](https://img.shields.io/badge/cdk--constructs-experimental-important.svg?style=for-the-badge) > The APIs of higher level constructs in this module are experimental and under active development. They are subject to non-backward compatible changes or removal in any future version. These are not subject to the [Semantic Versioning](https://semver.org/) model and breaking changes will be announced in the release notes. This means that while you may use them, you may need to update your source code when upgrading to a newer version of this package. ---

This module is part of the AWS Cloud Development Kit project.

Database

A Database is a logical grouping of Tables in the Glue Catalog.

# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
glue.Database(stack, "MyDatabase",
    database_name="my_database"
)

Table

A Glue table describes a table of data in S3: its structure (column names and types), location of data (S3 objects with a common prefix in a S3 bucket), and format for the files (Json, Avro, Parquet, etc.):

# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
glue.Table(stack, "MyTable",
    database=my_database,
    table_name="my_table",
    columns=[{
        "name": "col1",
        "type": glue.Schema.STRING
    }, {
        "name": "col2",
        "type": glue.Schema.array(Schema.STRING),
        "comment": "col2 is an array of strings"
    }],
    data_format=glue.DataFormat.JSON
)

By default, a S3 bucket will be created to store the table’s data but you can manually pass the bucket and s3Prefix:

# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
glue.Table(stack, "MyTable",
    bucket=my_bucket,
    s3_prefix="my-table/", ...
)

By default, an S3 bucket will be created to store the table’s data and stored in the bucket root. You can also manually pass the bucket and s3Prefix:

Partitions

To improve query performance, a table can specify partitionKeys on which data is stored and queried separately. For example, you might partition a table by year and month to optimize queries based on a time window:

# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
glue.Table(stack, "MyTable",
    database=my_database,
    table_name="my_table",
    columns=[{
        "name": "col1",
        "type": glue.Schema.STRING
    }],
    partition_keys=[{
        "name": "year",
        "type": glue.Schema.SMALL_INT
    }, {
        "name": "month",
        "type": glue.Schema.SMALL_INT
    }],
    data_format=glue.DataFormat.JSON
)

Encryption

You can enable encryption on a Table’s data:

  • Unencrypted - files are not encrypted. The default encryption setting.

  • S3Managed - Server side encryption (SSE-S3) with an Amazon S3-managed key.

# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
glue.Table(stack, "MyTable",
    encryption=glue.TableEncryption.S3_MANAGED, ...
)
  • Kms - Server-side encryption (SSE-KMS) with an AWS KMS Key managed by the account owner.

# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
# KMS key is created automatically
glue.Table(stack, "MyTable",
    encryption=glue.TableEncryption.KMS, ...
)

# with an explicit KMS key
glue.Table(stack, "MyTable",
    encryption=glue.TableEncryption.KMS,
    encryption_key=kms.Key(stack, "MyKey"), ...
)
  • KmsManaged - Server-side encryption (SSE-KMS), like Kms, except with an AWS KMS Key managed by the AWS Key Management Service.

# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
glue.Table(stack, "MyTable",
    encryption=glue.TableEncryption.KMS_MANAGED, ...
)
  • ClientSideKms - Client-side encryption (CSE-KMS) with an AWS KMS Key managed by the account owner.

# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
# KMS key is created automatically
glue.Table(stack, "MyTable",
    encryption=glue.TableEncryption.CLIENT_SIDE_KMS, ...
)

# with an explicit KMS key
glue.Table(stack, "MyTable",
    encryption=glue.TableEncryption.CLIENT_SIDE_KMS,
    encryption_key=kms.Key(stack, "MyKey"), ...
)

Note: you cannot provide a ``Bucket`` when creating the ``Table`` if you wish to use server-side encryption (``KMS``, ``KMS_MANAGED`` or ``S3_MANAGED``).

Types

A table’s schema is a collection of columns, each of which have a name and a type. Types are recursive structures, consisting of primitive and complex types:

# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
glue.Table(stack, "MyTable",
    columns=[{
        "name": "primitive_column",
        "type": glue.Schema.STRING
    }, {
        "name": "array_column",
        "type": glue.Schema.array(glue.Schema.INTEGER),
        "comment": "array<integer>"
    }, {
        "name": "map_column",
        "type": glue.Schema.map(glue.Schema.STRING, glue.Schema.TIMESTAMP),
        "comment": "map<string,string>"
    }, {
        "name": "struct_column",
        "type": glue.Schema.struct([
            name="nested_column",
            type=glue.Schema.DATE,
            comment="nested comment"
        ]),
        "comment": "struct<nested_column:date COMMENT 'nested comment'>"
    }], ...
)

Primitives

Numeric

Name

Type

Comments

FLOAT

Constant

A 32-bit single-precision floating point number

INTEGER

Constant

A 32-bit signed value in two’s complement format, with a minimum value of -2^31 and a maximum value of 2^31-1

DOUBLE

Constant

A 64-bit double-precision floating point number

BIG_INT

Constant

A 64-bit signed INTEGER in two’s complement format, with a minimum value of -2^63 and a maximum value of 2^63 -1

SMALL_INT

Constant

A 16-bit signed INTEGER in two’s complement format, with a minimum value of -2^15 and a maximum value of 2^15-1

TINY_INT

Constant

A 8-bit signed INTEGER in two’s complement format, with a minimum value of -2^7 and a maximum value of 2^7-1

Date and time

Name

Type

Comments

DATE

Constant

A date in UNIX format, such as YYYY-MM-DD.

TIMESTAMP

Constant

Date and time instant in the UNiX format, such as yyyy-mm-dd hh:mm:ss[.f…]. For example, TIMESTAMP ‘2008-09-15 03:04:05.324’. This format uses the session time zone.

String

Name

Type

Comments

STRING

Constant

A string literal enclosed in single or double quotes

decimal(precision: number, scale?: number)

Function

precision is the total number of digits. scale (optional) is the number of digits in fractional part with a default of 0. For example, use these type definitions: decimal(11,5), decimal(15)

char(length: number)

Function

Fixed length character data, with a specified length between 1 and 255, such as char(10)

varchar(length: number)

Function

Variable length character data, with a specified length between 1 and 65535, such as varchar(10)

Miscellaneous

Name

Type

Comments

BOOLEAN

Constant

Values are true and false

BINARY

Constant

Value is in binary

Complex

Name

Type

Comments

array(itemType: Type)

Function

An array of some other type

map(keyType: Type, valueType: Type)

Function

A map of some primitive key type to any value type

struct(collumns: Column[])

Function

Nested structure containing individually named and typed collumns