Programming AWS Glue ETL scripts in Scala

You can find Scala code examples and utilities for AWS Glue in the AWS Glue samples repository on the GitHub website.

AWS Glue supports an extension of the PySpark Scala dialect for scripting extract, transform, and load (ETL) jobs. The following sections describe how to use the AWS Glue Scala library and the AWS Glue API in ETL scripts, and provide reference documentation for the library.

Contents

• Using Scala
  • Testing on a DevEndpoint notebook
  • Testing on a DevEndpoint REPL
• Scala script example
• Scala API list
  • com.amazonaws.services.glue
  • com.amazonaws.services.glue.ml
  • com.amazonaws.services.glue.dq
  • com.amazonaws.services.glue.types
  • com.amazonaws.services.glue.util
  • ChoiceOption
    • ChoiceOption trait
    • ChoiceOption object
      • Apply
    • ChoiceOptionWithResolver
    • MatchCatalogSchemaChoiceOption
  • DataSink
    • writeDynamicFrame
    • pyWriteDynamicFrame
    • writeDataFrame
    • pyWriteDataFrame
    • setCatalogInfo
    • supportsFormat
    • setFormat
    • withFormat
    • setAccumulableSize
    • getOutputErrorRecordsAccumulable
    • errorsAsDynamicFrame
    • DataSink object
      • recordMetrics
  • DataSource trait
  • DynamicFrame
    • DynamicFrame class
      • errorsCount
      • applyMapping
      • assertErrorThreshold
      • Count
      • dropField
      • dropFields
      • dropNulls
      • errorsAsDynamicFrame
      • Filter
      • getName
      • getNumPartitions
      • getSchemaIfComputed
      • isSchemaComputed
      • javaToPython
      • Join
      • Map
      • mergeDynamicFrames
      • printSchema
      • recomputeSchema
      • Relationalize
      • renameField
      • Repartition
      • resolveChoice
      • Schema
      • selectField
      • selectFields
      • Show
      • SimplifyDDBJson
      • Spigot
      • splitFields
      • Def splitRows
      • stageErrorsCount
      • toDF
      • Unbox
      • Unnest
      • unnestDDBJson
      • withFrameSchema
      • Def withName
      • withTransformationContext
    • DynamicFrame object
      • Def apply
      • Def emptyDynamicFrame
      • Def fromPythonRDD
      • Def ignoreErrors
      • Def inlineErrors
      • Def newFrameWithErrors
  • DynamicRecord
    • addField
    • dropField
    • setError
    • isError
    • getError
    • clearError
    • Write
    • readFields
    • Clone
    • Schema
    • getRoot
    • toJson
    • getFieldNode
    • getField
    • hashCode
    • Equals
    • DynamicRecord object
      • Apply
    • RecordTraverser trait
  • GlueContext
    • addIngestionTimeColumns
    • createDataFrameFromOptions
    • forEachBatch
    • getCatalogSink
    • getCatalogSource
    • getJDBCSink
    • getSink
    • getSinkWithFormat
    • getSource
    • getSourceWithFormat
    • getSparkSession
    • startTransaction
    • commitTransaction
    • cancelTransaction
    • this
    • this
    • this
  • MappingSpec
    • MappingSpec case class
    • MappingSpec object
    • orderingByTarget
    • Apply
    • Apply
    • Apply
  • ResolveSpec
    • ResolveSpec object
      • Def
      • Def
    • ResolveSpec case class
      • Def methods
  • ArrayNode
    • ArrayNode case class
      • Def methods
  • BinaryNode
    • BinaryNode case class
      • Val fields
      • Def methods
  • BooleanNode
    • BooleanNode case class
      • Val fields
      • Def methods
  • ByteNode
    • ByteNode case class
      • Val fields
      • Def methods
  • DateNode
    • DateNode case class
      • Val fields
      • Def methods
  • DecimalNode
    • DecimalNode case class
      • Val fields
      • Def methods
  • DoubleNode
    • DoubleNode case class
      • Val fields
      • Def methods
  • DynamicNode
    • DynamicNode class
      • Def methods
    • DynamicNode object
      • Def methods
  • EvaluateDataQuality
    • apply
    • Example
  • FloatNode
    • FloatNode case class
      • Val fields
      • Def methods
  • FillMissingValues
    • Apply
  • FindMatches
    • Apply
  • FindIncrementalMatches
    • Apply
  • IntegerNode
    • IntegerNode case class
      • Val fields
      • Def methods
  • LongNode
    • LongNode case class
      • Val fields
      • Def methods
  • MapLikeNode
    • MapLikeNode class
      • Def methods
  • MapNode
    • MapNode case class
      • Def methods
  • NullNode
    • NullNode class
    • NullNode case object
  • ObjectNode
    • ObjectNode object
      • Def methods
    • ObjectNode case class
      • Def methods
  • ScalarNode
    • ScalarNode class
      • Def methods
    • ScalarNode object
      • Def methods
  • ShortNode
    • ShortNode case class
      • Val fields
      • Def methods
  • StringNode
    • StringNode case class
      • Val fields
      • Def methods
  • TimestampNode
    • TimestampNode case class
      • Val fields
      • Def methods
  • GlueArgParser
    • GlueArgParser object
      • Def methods
  • Job
    • Job object
      • Def methods