Package software.amazon.awscdk.services.glue.alpha
AWS Glue Construct Library
---
The APIs of higher level constructs in this module are experimental and under active development. They are subject to non-backward compatible changes or removal in any future version. These are not subject to the Semantic Versioning model and breaking changes will be announced in the release notes. This means that while you may use them, you may need to update your source code when upgrading to a newer version of this package.
This module is part of the AWS Cloud Development Kit project.
Job
A Job
encapsulates a script that connects to data sources, processes them, and then writes output to a data target.
There are 3 types of jobs supported by AWS Glue: Spark ETL, Spark Streaming, and Python Shell jobs.
The glue.JobExecutable
allows you to specify the type of job, the language to use and the code assets required by the job.
glue.Code
allows you to refer to the different code assets required by the job, either from an existing S3 location or from a local file path.
glue.ExecutionClass
allows you to specify FLEX
or STANDARD
. FLEX
is appropriate for non-urgent jobs such as pre-production jobs, testing, and one-time data loads.
Spark Jobs
These jobs run in an Apache Spark environment managed by AWS Glue.
ETL Jobs
An ETL job processes data in batches using Apache Spark.
Bucket bucket; Job.Builder.create(this, "ScalaSparkEtlJob") .executable(JobExecutable.scalaEtl(ScalaJobExecutableProps.builder() .glueVersion(GlueVersion.V4_0) .script(Code.fromBucket(bucket, "src/com/example/HelloWorld.scala")) .className("com.example.HelloWorld") .extraJars(List.of(Code.fromBucket(bucket, "jars/HelloWorld.jar"))) .build())) .workerType(WorkerType.G_8X) .description("an example Scala ETL job") .build();
Streaming Jobs
A Streaming job is similar to an ETL job, except that it performs ETL on data streams. It uses the Apache Spark Structured Streaming framework. Some Spark job features are not available to streaming ETL jobs.
Job.Builder.create(this, "PythonSparkStreamingJob") .executable(JobExecutable.pythonStreaming(PythonSparkJobExecutableProps.builder() .glueVersion(GlueVersion.V4_0) .pythonVersion(PythonVersion.THREE) .script(Code.fromAsset(join(__dirname, "job-script", "hello_world.py"))) .build())) .description("an example Python Streaming job") .build();
Python Shell Jobs
A Python shell job runs Python scripts as a shell and supports a Python version that depends on the AWS Glue version you are using. This can be used to schedule and run tasks that don't require an Apache Spark environment. Currently, three flavors are supported:
- PythonVersion.TWO (2.7; EOL)
- PythonVersion.THREE (3.6)
- PythonVersion.THREE_NINE (3.9)
Bucket bucket; Job.Builder.create(this, "PythonShellJob") .executable(JobExecutable.pythonShell(PythonShellExecutableProps.builder() .glueVersion(GlueVersion.V1_0) .pythonVersion(PythonVersion.THREE) .script(Code.fromBucket(bucket, "script.py")) .build())) .description("an example Python Shell job") .build();
Ray Jobs
These jobs run in a Ray environment managed by AWS Glue.
Job.Builder.create(this, "RayJob") .executable(JobExecutable.pythonRay(PythonRayExecutableProps.builder() .glueVersion(GlueVersion.V4_0) .pythonVersion(PythonVersion.THREE_NINE) .runtime(Runtime.RAY_TWO_FOUR) .script(Code.fromAsset(join(__dirname, "job-script", "hello_world.py"))) .build())) .workerType(WorkerType.Z_2X) .workerCount(2) .description("an example Ray job") .build();
Enable Spark UI
Enable Spark UI setting the sparkUI
property.
Job.Builder.create(this, "EnableSparkUI") .jobName("EtlJobWithSparkUIPrefix") .sparkUI(SparkUIProps.builder() .enabled(true) .build()) .executable(JobExecutable.pythonEtl(PythonSparkJobExecutableProps.builder() .glueVersion(GlueVersion.V3_0) .pythonVersion(PythonVersion.THREE) .script(Code.fromAsset(join(__dirname, "job-script", "hello_world.py"))) .build())) .build();
The sparkUI
property also allows the specification of an s3 bucket and a bucket prefix.
See documentation for more information on adding jobs in Glue.
Connection
A Connection
allows Glue jobs, crawlers and development endpoints to access certain types of data stores. For example, to create a network connection to connect to a data source within a VPC:
SecurityGroup securityGroup; Subnet subnet; Connection.Builder.create(this, "MyConnection") .type(ConnectionType.NETWORK) // The security groups granting AWS Glue inbound access to the data source within the VPC .securityGroups(List.of(securityGroup)) // The VPC subnet which contains the data source .subnet(subnet) .build();
For RDS Connection
by JDBC, it is recommended to manage credentials using AWS Secrets Manager. To use Secret, specify SECRET_ID
in properties
like the following code. Note that in this case, the subnet must have a route to the AWS Secrets Manager VPC endpoint or to the AWS Secrets Manager endpoint through a NAT gateway.
SecurityGroup securityGroup; Subnet subnet; DatabaseCluster db; Connection.Builder.create(this, "RdsConnection") .type(ConnectionType.JDBC) .securityGroups(List.of(securityGroup)) .subnet(subnet) .properties(Map.of( "JDBC_CONNECTION_URL", String.format("jdbc:mysql://%s/databasename", db.getClusterEndpoint().getSocketAddress()), "JDBC_ENFORCE_SSL", "false", "SECRET_ID", db.getSecret().getSecretName())) .build();
If you need to use a connection type that doesn't exist as a static member on ConnectionType
, you can instantiate a ConnectionType
object, e.g: new glue.ConnectionType('NEW_TYPE')
.
See Adding a Connection to Your Data Store and Connection Structure documentation for more information on the supported data stores and their configurations.
SecurityConfiguration
A SecurityConfiguration
is a set of security properties that can be used by AWS Glue to encrypt data at rest.
SecurityConfiguration.Builder.create(this, "MySecurityConfiguration") .cloudWatchEncryption(CloudWatchEncryption.builder() .mode(CloudWatchEncryptionMode.KMS) .build()) .jobBookmarksEncryption(JobBookmarksEncryption.builder() .mode(JobBookmarksEncryptionMode.CLIENT_SIDE_KMS) .build()) .s3Encryption(S3Encryption.builder() .mode(S3EncryptionMode.KMS) .build()) .build();
By default, a shared KMS key is created for use with the encryption configurations that require one. You can also supply your own key for each encryption config, for example, for CloudWatch encryption:
Key key; SecurityConfiguration.Builder.create(this, "MySecurityConfiguration") .cloudWatchEncryption(CloudWatchEncryption.builder() .mode(CloudWatchEncryptionMode.KMS) .kmsKey(key) .build()) .build();
See documentation for more info for Glue encrypting data written by Crawlers, Jobs, and Development Endpoints.
Database
A Database
is a logical grouping of Tables
in the Glue Catalog.
Database.Builder.create(this, "MyDatabase") .databaseName("my_database") .description("my_database_description") .build();
Table
A Glue table describes a table of data in S3: its structure (column names and types), location of data (S3 objects with a common prefix in a S3 bucket), and format for the files (Json, Avro, Parquet, etc.):
Database myDatabase; S3Table.Builder.create(this, "MyTable") .database(myDatabase) .columns(List.of(Column.builder() .name("col1") .type(Schema.STRING) .build(), Column.builder() .name("col2") .type(Schema.array(Schema.STRING)) .comment("col2 is an array of strings") .build())) .dataFormat(DataFormat.JSON) .build();
By default, a S3 bucket will be created to store the table's data but you can manually pass the bucket
and s3Prefix
:
Bucket myBucket; Database myDatabase; S3Table.Builder.create(this, "MyTable") .bucket(myBucket) .s3Prefix("my-table/") // ... .database(myDatabase) .columns(List.of(Column.builder() .name("col1") .type(Schema.STRING) .build())) .dataFormat(DataFormat.JSON) .build();
Glue tables can be configured to contain user-defined properties, to describe the physical storage of table data, through the storageParameters
property:
Database myDatabase; S3Table.Builder.create(this, "MyTable") .storageParameters(List.of(StorageParameter.skipHeaderLineCount(1), StorageParameter.compressionType(CompressionType.GZIP), StorageParameter.custom("separatorChar", ","))) // ... .database(myDatabase) .columns(List.of(Column.builder() .name("col1") .type(Schema.STRING) .build())) .dataFormat(DataFormat.JSON) .build();
Glue tables can also be configured to contain user-defined table properties through the parameters
property:
Database myDatabase; S3Table.Builder.create(this, "MyTable") .parameters(Map.of( "key1", "val1", "key2", "val2")) .database(myDatabase) .columns(List.of(Column.builder() .name("col1") .type(Schema.STRING) .build())) .dataFormat(DataFormat.JSON) .build();
Partition Keys
To improve query performance, a table can specify partitionKeys
on which data is stored and queried separately. For example, you might partition a table by year
and month
to optimize queries based on a time window:
Database myDatabase; S3Table.Builder.create(this, "MyTable") .database(myDatabase) .columns(List.of(Column.builder() .name("col1") .type(Schema.STRING) .build())) .partitionKeys(List.of(Column.builder() .name("year") .type(Schema.SMALL_INT) .build(), Column.builder() .name("month") .type(Schema.SMALL_INT) .build())) .dataFormat(DataFormat.JSON) .build();
Partition Indexes
Another way to improve query performance is to specify partition indexes. If no partition indexes are present on the table, AWS Glue loads all partitions of the table and filters the loaded partitions using the query expression. The query takes more time to run as the number of partitions increase. With an index, the query will try to fetch a subset of the partitions instead of loading all partitions of the table.
The keys of a partition index must be a subset of the partition keys of the table. You can have a
maximum of 3 partition indexes per table. To specify a partition index, you can use the partitionIndexes
property:
Database myDatabase; S3Table.Builder.create(this, "MyTable") .database(myDatabase) .columns(List.of(Column.builder() .name("col1") .type(Schema.STRING) .build())) .partitionKeys(List.of(Column.builder() .name("year") .type(Schema.SMALL_INT) .build(), Column.builder() .name("month") .type(Schema.SMALL_INT) .build())) .partitionIndexes(List.of(PartitionIndex.builder() .indexName("my-index") // optional .keyNames(List.of("year")) .build())) // supply up to 3 indexes .dataFormat(DataFormat.JSON) .build();
Alternatively, you can call the addPartitionIndex()
function on a table:
Table myTable; myTable.addPartitionIndex(PartitionIndex.builder() .indexName("my-index") .keyNames(List.of("year")) .build());
Partition Filtering
If you have a table with a large number of partitions that grows over time, consider using AWS Glue partition indexing and filtering.
Database myDatabase; S3Table.Builder.create(this, "MyTable") .database(myDatabase) .columns(List.of(Column.builder() .name("col1") .type(Schema.STRING) .build())) .partitionKeys(List.of(Column.builder() .name("year") .type(Schema.SMALL_INT) .build(), Column.builder() .name("month") .type(Schema.SMALL_INT) .build())) .dataFormat(DataFormat.JSON) .enablePartitionFiltering(true) .build();
Glue Connections
Glue connections allow external data connections to third party databases and data warehouses. However, these connections can also be assigned to Glue Tables, allowing you to query external data sources using the Glue Data Catalog.
Whereas S3Table
will point to (and if needed, create) a bucket to store the tables' data, ExternalTable
will point to an existing table in a data source. For example, to create a table in Glue that points to a table in Redshift:
Connection myConnection; Database myDatabase; ExternalTable.Builder.create(this, "MyTable") .connection(myConnection) .externalDataLocation("default_db_public_example") // A table in Redshift // ... .database(myDatabase) .columns(List.of(Column.builder() .name("col1") .type(Schema.STRING) .build())) .dataFormat(DataFormat.JSON) .build();
Encryption
You can enable encryption on a Table's data:
- S3Managed - (default) Server side encryption (
SSE-S3
) with an Amazon S3-managed key.
Database myDatabase; S3Table.Builder.create(this, "MyTable") .encryption(TableEncryption.S3_MANAGED) // ... .database(myDatabase) .columns(List.of(Column.builder() .name("col1") .type(Schema.STRING) .build())) .dataFormat(DataFormat.JSON) .build();
- Kms - Server-side encryption (
SSE-KMS
) with an AWS KMS Key managed by the account owner.
Database myDatabase; // KMS key is created automatically // KMS key is created automatically S3Table.Builder.create(this, "MyTable") .encryption(TableEncryption.KMS) // ... .database(myDatabase) .columns(List.of(Column.builder() .name("col1") .type(Schema.STRING) .build())) .dataFormat(DataFormat.JSON) .build(); // with an explicit KMS key // with an explicit KMS key S3Table.Builder.create(this, "MyTable") .encryption(TableEncryption.KMS) .encryptionKey(new Key(this, "MyKey")) // ... .database(myDatabase) .columns(List.of(Column.builder() .name("col1") .type(Schema.STRING) .build())) .dataFormat(DataFormat.JSON) .build();
- KmsManaged - Server-side encryption (
SSE-KMS
), likeKms
, except with an AWS KMS Key managed by the AWS Key Management Service.
Database myDatabase; S3Table.Builder.create(this, "MyTable") .encryption(TableEncryption.KMS_MANAGED) // ... .database(myDatabase) .columns(List.of(Column.builder() .name("col1") .type(Schema.STRING) .build())) .dataFormat(DataFormat.JSON) .build();
- ClientSideKms - Client-side encryption (
CSE-KMS
) with an AWS KMS Key managed by the account owner.
Database myDatabase; // KMS key is created automatically // KMS key is created automatically S3Table.Builder.create(this, "MyTable") .encryption(TableEncryption.CLIENT_SIDE_KMS) // ... .database(myDatabase) .columns(List.of(Column.builder() .name("col1") .type(Schema.STRING) .build())) .dataFormat(DataFormat.JSON) .build(); // with an explicit KMS key // with an explicit KMS key S3Table.Builder.create(this, "MyTable") .encryption(TableEncryption.CLIENT_SIDE_KMS) .encryptionKey(new Key(this, "MyKey")) // ... .database(myDatabase) .columns(List.of(Column.builder() .name("col1") .type(Schema.STRING) .build())) .dataFormat(DataFormat.JSON) .build();
Note: you cannot provide a Bucket
when creating the S3Table
if you wish to use server-side encryption (KMS
, KMS_MANAGED
or S3_MANAGED
).
Types
A table's schema is a collection of columns, each of which have a name
and a type
. Types are recursive structures, consisting of primitive and complex types:
Database myDatabase; S3Table.Builder.create(this, "MyTable") .columns(List.of(Column.builder() .name("primitive_column") .type(Schema.STRING) .build(), Column.builder() .name("array_column") .type(Schema.array(Schema.INTEGER)) .comment("array<integer>") .build(), Column.builder() .name("map_column") .type(Schema.map(Schema.STRING, Schema.TIMESTAMP)) .comment("map<string,string>") .build(), Column.builder() .name("struct_column") .type(Schema.struct(List.of(Column.builder() .name("nested_column") .type(Schema.DATE) .comment("nested comment") .build()))) .comment("struct<nested_column:date COMMENT 'nested comment'>") .build())) // ... .database(myDatabase) .dataFormat(DataFormat.JSON) .build();
Primitives
Numeric
| Name | Type | Comments | |----------- |---------- |------------------------------------------------------------------------------------------------------------------ | | FLOAT | Constant | A 32-bit single-precision floating point number | | INTEGER | Constant | A 32-bit signed value in two's complement format, with a minimum value of -2^31 and a maximum value of 2^31-1 | | DOUBLE | Constant | A 64-bit double-precision floating point number | | BIG_INT | Constant | A 64-bit signed INTEGER in two’s complement format, with a minimum value of -2^63 and a maximum value of 2^63 -1 | | SMALL_INT | Constant | A 16-bit signed INTEGER in two’s complement format, with a minimum value of -2^15 and a maximum value of 2^15-1 | | TINY_INT | Constant | A 8-bit signed INTEGER in two’s complement format, with a minimum value of -2^7 and a maximum value of 2^7-1 |
Date and time
| Name | Type | Comments | |----------- |---------- |------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | DATE | Constant | A date in UNIX format, such as YYYY-MM-DD. | | TIMESTAMP | Constant | Date and time instant in the UNiX format, such as yyyy-mm-dd hh:mm:ss[.f...]. For example, TIMESTAMP '2008-09-15 03:04:05.324'. This format uses the session time zone. |
String
| Name | Type | Comments |
|-------------------------------------------- |---------- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| STRING | Constant | A string literal enclosed in single or double quotes |
| decimal(precision: number, scale?: number) | Function | precision
is the total number of digits. scale
(optional) is the number of digits in fractional part with a default of 0. For example, use these type definitions: decimal(11,5), decimal(15) |
| char(length: number) | Function | Fixed length character data, with a specified length between 1 and 255, such as char(10) |
| varchar(length: number) | Function | Variable length character data, with a specified length between 1 and 65535, such as varchar(10) |
Miscellaneous
| Name | Type | Comments |
|--------- |---------- |------------------------------- |
| BOOLEAN | Constant | Values are true
and false
|
| BINARY | Constant | Value is in binary |
Complex
| Name | Type | Comments | |------------------------------------- |---------- |------------------------------------------------------------------- | | array(itemType: Type) | Function | An array of some other type | | map(keyType: Type, valueType: Type) | Function | A map of some primitive key type to any value type | | struct(collumns: Column[]) | Function | Nested structure containing individually named and typed collumns |
Data Quality Ruleset
A DataQualityRuleset
specifies a data quality ruleset with DQDL rules applied to a specified AWS Glue table. For example, to create a data quality ruleset for a given table:
DataQualityRuleset.Builder.create(this, "MyDataQualityRuleset") .clientToken("client_token") .description("description") .rulesetName("ruleset_name") .rulesetDqdl("ruleset_dqdl") .tags(Map.of( "key1", "value1", "key2", "value2")) .targetTable(new DataQualityTargetTable("database_name", "table_name")) .build();
For more information, see AWS Glue Data Quality.
-
ClassDescription(experimental) Job Code from a local file.(experimental) Classification string given to tables with this data format.(experimental) CloudWatch Logs encryption configuration.A builder for
CloudWatchEncryption
An implementation forCloudWatchEncryption
(experimental) Encryption mode for CloudWatch Logs.(experimental) Represents a Glue Job's Code assets (an asset can be a scripts, a jar, a python file or any other file).(experimental) Result of bindingCode
into aJob
.A builder forCodeConfig
An implementation forCodeConfig
(experimental) A column of a table.A builder forColumn
An implementation forColumn
(experimental) Identifies if the file contains less or more values for a row than the number of columns specified in the external table definition.(experimental) The compression type.(experimental) An AWS Glue connection to a data source.(experimental) A fluent builder forConnection
.(experimental) Base Connection Options.A builder forConnectionOptions
An implementation forConnectionOptions
(experimental) Construction properties forConnection
.A builder forConnectionProps
An implementation forConnectionProps
(experimental) The type of the glue connection.(experimental) Properties for enabling Continuous Logging for Glue Jobs.A builder forContinuousLoggingProps
An implementation forContinuousLoggingProps
(experimental) A Glue database.(experimental) A fluent builder forDatabase
.Example:A builder forDatabaseProps
An implementation forDatabaseProps
(experimental) Defines the input/output formats and ser/de for a single DataFormat.(experimental) A fluent builder forDataFormat
.(experimental) Properties of a DataFormat instance.A builder forDataFormatProps
An implementation forDataFormatProps
(experimental) A Glue Data Quality ruleset.(experimental) A fluent builder forDataQualityRuleset
.(experimental) Construction properties forDataQualityRuleset
.A builder forDataQualityRulesetProps
An implementation forDataQualityRulesetProps
(experimental) Properties of a DataQualityTargetTable.(experimental) The ExecutionClass whether the job is run with a standard or flexible execution class.(experimental) A Glue table that targets an external data location (e.g.(experimental) A fluent builder forExternalTable
.Example:A builder forExternalTableProps
An implementation forExternalTableProps
(experimental) AWS Glue version determines the versions of Apache Spark and Python that are available to the job.(experimental) Interface representing a created or an importedConnection
.Internal default implementation forIConnection
.A proxy class which represents a concrete javascript instance of this type.Internal default implementation forIDatabase
.A proxy class which represents a concrete javascript instance of this type.Internal default implementation forIDataQualityRuleset
.A proxy class which represents a concrete javascript instance of this type.(experimental) Interface representing a created or an importedJob
.Internal default implementation forIJob
.A proxy class which represents a concrete javascript instance of this type.(experimental) Absolute class name of the HadoopInputFormat
to use when reading table files.(experimental) Specifies the action to perform when query results contain invalid UTF-8 character values.(experimental) Interface representing a created or an importedSecurityConfiguration
.Internal default implementation forISecurityConfiguration
.A proxy class which represents a concrete javascript instance of this type.Internal default implementation forITable
.A proxy class which represents a concrete javascript instance of this type.(experimental) A Glue Job.(experimental) A fluent builder forJob
.(experimental) Attributes for importingJob
.A builder forJobAttributes
An implementation forJobAttributes
(experimental) Job bookmarks encryption configuration.A builder forJobBookmarksEncryption
An implementation forJobBookmarksEncryption
(experimental) Encryption mode for Job Bookmarks.(experimental) The executable properties related to the Glue job's GlueVersion, JobType and code.(experimental) Result of binding aJobExecutable
into aJob
.A builder forJobExecutableConfig
An implementation forJobExecutableConfig
(experimental) Runtime language of the Glue job.(experimental) Construction properties forJob
.A builder forJobProps
An implementation forJobProps
(experimental) Job states emitted by Glue to CloudWatch Events.(experimental) The job type.(experimental) The Glue CloudWatch metric type.(experimental) Specifies the action to perform when ORC data contains an integer (for example, BIGINT or int64) that is larger than the column definition (for example, SMALLINT or int16).(experimental) Specifies how to map columns when the table uses ORC data format.(experimental) Absolute class name of the HadoopOutputFormat
to use when writing table files.(experimental) Properties of a Partition Index.A builder forPartitionIndex
An implementation forPartitionIndex
(experimental) Props for creating a Python Ray job executable.A builder forPythonRayExecutableProps
An implementation forPythonRayExecutableProps
(experimental) Props for creating a Python shell job executable.A builder forPythonShellExecutableProps
An implementation forPythonShellExecutableProps
(experimental) Props for creating a Python Spark (ETL or Streaming) job executable.A builder forPythonSparkJobExecutableProps
An implementation forPythonSparkJobExecutableProps
(experimental) Python version.(experimental) AWS Glue runtime determines the runtime engine of the job.(experimental) Glue job Code from an S3 bucket.(experimental) S3 encryption configuration.A builder forS3Encryption
An implementation forS3Encryption
(experimental) Encryption mode for S3.(experimental) A Glue table that targets a S3 dataset.(experimental) A fluent builder forS3Table
.Example:A builder forS3TableProps
An implementation forS3TableProps
(experimental) Props for creating a Scala Spark (ETL or Streaming) job executable.A builder forScalaJobExecutableProps
An implementation forScalaJobExecutableProps
Example:(experimental) A security configuration is a set of security properties that can be used by AWS Glue to encrypt data at rest.(experimental) A fluent builder forSecurityConfiguration
.(experimental) Constructions properties ofSecurityConfiguration
.A builder forSecurityConfigurationProps
An implementation forSecurityConfigurationProps
(experimental) Serialization library to use when serializing/deserializing (SerDe) table records.(experimental) The Spark UI logging location.A builder forSparkUILoggingLocation
An implementation forSparkUILoggingLocation
(experimental) Properties for enabling Spark UI monitoring feature for Spark-based Glue jobs.A builder forSparkUIProps
An implementation forSparkUIProps
(experimental) A storage parameter.(experimental) The storage parameter keys that are currently known, this list is not exhaustive and other keys may be used.(experimental) Specifies how to handle data being loaded that exceeds the length of the data type defined for columns containing VARBYTE data.(experimental) Specifies how to handle data being loaded that exceeds the length of the data type defined for columns containing VARCHAR, CHAR, or string data.Deprecated.(experimental) A fluent builder forTable
.Example:A builder forTableAttributes
An implementation forTableAttributes
(experimental) A Glue table.Example:A builder forTableBaseProps
An implementation forTableBaseProps
(experimental) Encryption options for a Table.Example:A builder forTableProps
An implementation forTableProps
(experimental) Represents a type of a column in a table schema.A builder forType
An implementation forType
(experimental) The type of predefined worker that is allocated when a job runs.(experimental) Specifies how to handle data being loaded that exceeds the length of the data type defined for columns containing VARCHAR, CHAR, or string data.
S3Table
instead.