Menu
AWS Glue
Developer Guide

AWS Glue API for Generating ETL Scripts

Data Types

CodeGenNode Structure

Represents a node in a directed acyclic graph (DAG)

Fields

  • Id – String, matching the Identifier string pattern. Required.

    A node identifier that is unique within the node's graph.

  • NodeType – String. Required.

    The type of node this is.

  • Args – An array of CodeGenNodeArgs. Required.

    Properties of the node, in the form of name-value pairs.

  • LineNumber – Number (integer).

    The line number of the node.

CodeGenNodeArg Structure

An argument or property of a node.

Fields

  • Name – String. Required.

    The name of the argument or property.

  • Value – String. Required.

    The value of the argument or property.

  • Param – Boolean.

    True if the value is used as a parameter.

CodeGenEdge Structure

Represents a directional edge in a directed acyclic graph (DAG).

Fields

  • Source – String, matching the Identifier string pattern. Required.

    The ID of the node at which the edge starts.

  • Target – String, matching the Identifier string pattern. Required.

    The ID of the node at which the edge ends.

  • TargetParameter – String.

    The target of the edge.

Location Structure

The location of resources.

Fields

CatalogEntry Structure

Specifies a table definition in the Data Catalog.

Fields

MappingEntry Structure

Defines a mapping.

Fields

  • SourceTable – String.

    The name of the source table.

  • SourcePath – String.

    The source path.

  • SourceType – String.

    The source type.

  • TargetTable – String.

    The target table.

  • TargetPath – String.

    The target path.

  • TargetType – String.

    The target type.

Operations

CreateScript Action (Python: create_script)

Transforms a directed acyclic graph (DAG) into a Python script.

Request

  • DagNodes – An array of CodeGenNodes.

    A list of the nodes in the DAG.

  • DagEdges – An array of CodeGenEdges.

    A list of the edges in the DAG.

Response

  • PythonScript – String.

    The Python script generated from the DAG.

Errors

  • InvalidInputException

  • InternalServiceException

  • OperationTimeoutException

GetDataflowGraph Action (Python: get_dataflow_graph)

Transforms a Python script into a directed acyclic graph (DAG).

Request

  • PythonScript – String.

    The Python script to transform.

Response

  • DagNodes – An array of CodeGenNodes.

    A list of the nodes in the resulting DAG.

  • DagEdges – An array of CodeGenEdges.

    A list of the edges in the resulting DAG.

Errors

  • InvalidInputException

  • InternalServiceException

  • OperationTimeoutException

GetMapping Action (Python: get_mapping)

Creates mappings.

Request

  • Source – A CatalogEntry object. Required.

    Specifies the source table.

  • Sinks – An array of CatalogEntrys.

    A list of target tables.

  • Location – A Location object.

    Parameters for the mapping.

Response

  • Mapping – An array of MappingEntrys. Required.

    A list of mappings to the specified targets.

Errors

  • InvalidInputException

  • InternalServiceException

  • OperationTimeoutException

  • EntityNotFoundException

GetPlan Action (Python: get_plan)

Gets a Python script to perform a specified mapping.

Request

  • Mapping – An array of MappingEntrys. Required.

    The list of mappings from a source table to target tables.

  • Source – A CatalogEntry object. Required.

    The source table.

  • Sinks – An array of CatalogEntrys.

    The target tables.

  • Location – A Location object.

    Parameters for the mapping.

Response

  • PythonScript – String.

    A python script to perform the mapping.

Errors

  • InvalidInputException

  • InternalServiceException

  • OperationTimeoutException