Class PythonSparkJobExecutableProps
(experimental) Props for creating a Python Spark (ETL or Streaming) job executable.
Inheritance
Implements
Namespace: Amazon.CDK.AWS.Glue.Alpha
Assembly: Amazon.CDK.AWS.Glue.Alpha.dll
Syntax (csharp)
public class PythonSparkJobExecutableProps : Object, IPythonSparkJobExecutableProps
Syntax (vb)
Public Class PythonSparkJobExecutableProps
Inherits Object
Implements IPythonSparkJobExecutableProps
Remarks
Stability: Experimental
ExampleMetadata: infused
Examples
new Job(this, "PythonSparkStreamingJob", new JobProps {
Executable = JobExecutable.PythonStreaming(new PythonSparkJobExecutableProps {
GlueVersion = GlueVersion.V4_0,
PythonVersion = PythonVersion.THREE,
Script = Code.FromAsset(Join(__dirname, "job-script", "hello_world.py"))
}),
Description = "an example Python Streaming job"
});
Synopsis
Constructors
PythonSparkJobExecutableProps() |
Properties
ExtraFiles | (experimental) Additional files, such as configuration files that AWS Glue copies to the working directory of your script before executing it. |
ExtraJars | (experimental) Additional Java .jar files that AWS Glue adds to the Java classpath before executing your script. Only individual files are supported, directories are not supported. Equivalent to a job parameter |
ExtraJarsFirst | (experimental) Setting this value to true prioritizes the customer's extra JAR files in the classpath. |
ExtraPythonFiles | (experimental) Additional Python files that AWS Glue adds to the Python path before executing your script. |
GlueVersion | (experimental) Glue version. |
PythonVersion | (experimental) The Python version to use. |
Runtime | (experimental) Runtime. |
Script | (experimental) The script that executes a job. |
Constructors
PythonSparkJobExecutableProps()
public PythonSparkJobExecutableProps()
Properties
ExtraFiles
(experimental) Additional files, such as configuration files that AWS Glue copies to the working directory of your script before executing it.
public Code[] ExtraFiles { get; set; }
Property Value
Code[]
Remarks
Only individual files are supported, directories are not supported.
Equivalent to a job parameter --extra-files
.
Default: [] - no extra files are copied to the working directory
Stability: Experimental
See: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html
ExtraJars
(experimental) Additional Java .jar files that AWS Glue adds to the Java classpath before executing your script. Only individual files are supported, directories are not supported. Equivalent to a job parameter --extra-jars
.
public Code[] ExtraJars { get; set; }
Property Value
Code[]
Remarks
Default: [] - no extra jars are added to the classpath
Stability: Experimental
See: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html
ExtraJarsFirst
(experimental) Setting this value to true prioritizes the customer's extra JAR files in the classpath.
public Nullable<bool> ExtraJarsFirst { get; set; }
Property Value
System.Nullable<System.Boolean>
Remarks
Equivalent to a job parameter --user-jars-first
.
Default: false - priority is not given to user-provided jars
Stability: Experimental
See: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html
ExtraPythonFiles
(experimental) Additional Python files that AWS Glue adds to the Python path before executing your script.
public Code[] ExtraPythonFiles { get; set; }
Property Value
Code[]
Remarks
Only individual files are supported, directories are not supported.
Equivalent to a job parameter --extra-py-files
.
Default: - no extra python files and argument is not set
Stability: Experimental
See: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html
GlueVersion
(experimental) Glue version.
public GlueVersion GlueVersion { get; set; }
Property Value
Remarks
Stability: Experimental
See: https://docs.aws.amazon.com/glue/latest/dg/release-notes.html
PythonVersion
(experimental) The Python version to use.
public PythonVersion PythonVersion { get; set; }
Property Value
Remarks
Stability: Experimental
Runtime
(experimental) Runtime.
public Runtime Runtime { get; set; }
Property Value
Remarks
It is required for Ray jobs.
Stability: Experimental
Script
(experimental) The script that executes a job.
public Code Script { get; set; }
Property Value
Remarks
Stability: Experimental