Program AWS Glue ETL scripts in PySpark

You can find Python code examples and utilities for AWS Glue in the AWS Glue samples repository on the GitHub website.

Using Python with AWS Glue

AWS Glue supports an extension of the PySpark Python dialect for scripting extract, transform, and load (ETL) jobs. This section describes how to use Python in ETL scripts and with the AWS Glue API.

• Setting up to use Python with AWS Glue

• Calling AWS Glue APIs in Python

• Using Python libraries with AWS Glue

• AWS Glue Python code samples

AWS Glue PySpark extensions

AWS Glue has created the following extensions to the PySpark Python dialect.

• Accessing parameters using getResolvedOptions

• PySpark extension types

• DynamicFrame class

• DynamicFrameCollection class

• DynamicFrameWriter class

• DynamicFrameReader class

• GlueContext class

AWS Glue PySpark transforms

AWS Glue has created the following transform Classes to use in PySpark ETL operations.

• GlueTransform base class

• ApplyMapping class

• DropFields class

• DropNullFields class

• ErrorsAsDynamicFrame class

• FillMissingValues class

• Filter class

• FindIncrementalMatches class

• FindMatches class

• FlatMap class

• Join class

• Map class

• MapToCollection class

• mergeDynamicFrame

• Relationalize class

• RenameField class

• ResolveChoice class

• SelectFields class

• SelectFromCollection class

• Spigot class

• SplitFields class

• SplitRows class

• Unbox class

• UnnestFrame class