interface PythonSparkJobExecutableProps
Language | Type name |
---|---|
.NET | Amazon.CDK.AWS.Glue.Alpha.PythonSparkJobExecutableProps |
Go | github.com/aws/aws-cdk-go/awscdkgluealpha/v2#PythonSparkJobExecutableProps |
Java | software.amazon.awscdk.services.glue.alpha.PythonSparkJobExecutableProps |
Python | aws_cdk.aws_glue_alpha.PythonSparkJobExecutableProps |
TypeScript (source) | @aws-cdk/aws-glue-alpha ยป PythonSparkJobExecutableProps |
Props for creating a Python Spark (ETL or Streaming) job executable.
Example
new glue.Job(this, 'PythonSparkStreamingJob', {
executable: glue.JobExecutable.pythonStreaming({
glueVersion: glue.GlueVersion.V4_0,
pythonVersion: glue.PythonVersion.THREE,
script: glue.Code.fromAsset(path.join(__dirname, 'job-script', 'hello_world.py')),
}),
description: 'an example Python Streaming job',
});
Properties
Name | Type | Description |
---|---|---|
glue | Glue | Glue version. |
python | Python | The Python version to use. |
script | Code | The script that executes a job. |
extra | Code [] | Additional files, such as configuration files that AWS Glue copies to the working directory of your script before executing it. |
extra | Code [] | Additional Java .jar files that AWS Glue adds to the Java classpath before executing your script. Only individual files are supported, directories are not supported. Equivalent to a job parameter --extra-jars . |
extra | boolean | Setting this value to true prioritizes the customer's extra JAR files in the classpath. |
extra | Code [] | Additional Python files that AWS Glue adds to the Python path before executing your script. |
runtime? | Runtime | Runtime. |
glueVersion
Type:
Glue
Glue version.
See also: https://docs.aws.amazon.com/glue/latest/dg/release-notes.html
pythonVersion
Type:
Python
The Python version to use.
script
Type:
Code
The script that executes a job.
extraFiles?
Type:
Code
[]
(optional, default: [] - no extra files are copied to the working directory)
Additional files, such as configuration files that AWS Glue copies to the working directory of your script before executing it.
Only individual files are supported, directories are not supported.
Equivalent to a job parameter --extra-files
.
See also: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html
extraJars?
Type:
Code
[]
(optional, default: [] - no extra jars are added to the classpath)
Additional Java .jar files that AWS Glue adds to the Java classpath before executing your script. Only individual files are supported, directories are not supported. Equivalent to a job parameter --extra-jars
.
See also: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html
extraJarsFirst?
Type:
boolean
(optional, default: false - priority is not given to user-provided jars)
Setting this value to true prioritizes the customer's extra JAR files in the classpath.
Equivalent to a job parameter --user-jars-first
.
See also: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html
extraPythonFiles?
Type:
Code
[]
(optional, default: no extra python files and argument is not set)
Additional Python files that AWS Glue adds to the Python path before executing your script.
Only individual files are supported, directories are not supported.
Equivalent to a job parameter --extra-py-files
.
See also: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html
runtime?
Type:
Runtime
(optional)
Runtime.
It is required for Ray jobs.