PythonSparkJobExecutableProps

class aws_cdk.aws_glue_alpha.PythonSparkJobExecutableProps(*, glue_version, python_version, script, extra_files=None, extra_jars=None, extra_jars_first=None, extra_python_files=None, runtime=None)

Bases: object

(experimental) Props for creating a Python Spark (ETL or Streaming) job executable.

Parameters:

glue_version (GlueVersion) – (experimental) Glue version.
python_version (PythonVersion) – (experimental) The Python version to use.
script (Code) – (experimental) The script that executes a job.
extra_files (Optional[Sequence[Code]]) – (experimental) Additional files, such as configuration files that AWS Glue copies to the working directory of your script before executing it. Only individual files are supported, directories are not supported. Equivalent to a job parameter --extra-files. Default: [] - no extra files are copied to the working directory
extra_jars (Optional[Sequence[Code]]) – (experimental) Additional Java .jar files that AWS Glue adds to the Java classpath before executing your script. Only individual files are supported, directories are not supported. Equivalent to a job parameter --extra-jars. Default: [] - no extra jars are added to the classpath
extra_jars_first (Optional[bool]) – (experimental) Setting this value to true prioritizes the customer’s extra JAR files in the classpath. Equivalent to a job parameter --user-jars-first. Default: false - priority is not given to user-provided jars
extra_python_files (Optional[Sequence[Code]]) – (experimental) Additional Python files that AWS Glue adds to the Python path before executing your script. Only individual files are supported, directories are not supported. Equivalent to a job parameter --extra-py-files. Default: - no extra python files and argument is not set
runtime (Optional[Runtime]) – (experimental) Runtime. It is required for Ray jobs.

Stability:

experimental

ExampleMetadata:

infused

Example:

glue.Job(self, "PythonSparkStreamingJob",
    executable=glue.JobExecutable.python_streaming(
        glue_version=glue.GlueVersion.V4_0,
        python_version=glue.PythonVersion.THREE,
        script=glue.Code.from_asset(path.join(__dirname, "job-script", "hello_world.py"))
    ),
    description="an example Python Streaming job"
)

Attributes

extra_files

(experimental) Additional files, such as configuration files that AWS Glue copies to the working directory of your script before executing it.

Only individual files are supported, directories are not supported. Equivalent to a job parameter --extra-files.

Default:: [] - no extra files are copied to the working directory
See:: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html
Stability:: experimental

extra_jars

(experimental) Additional Java .jar files that AWS Glue adds to the Java classpath before executing your script. Only individual files are supported, directories are not supported. Equivalent to a job parameter --extra-jars.

Default:: [] - no extra jars are added to the classpath
See:: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html
Stability:: experimental

extra_jars_first

(experimental) Setting this value to true prioritizes the customer’s extra JAR files in the classpath.

Equivalent to a job parameter --user-jars-first.

Default:: false - priority is not given to user-provided jars
See:: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html
Stability:: experimental

extra_python_files

(experimental) Additional Python files that AWS Glue adds to the Python path before executing your script.

Only individual files are supported, directories are not supported. Equivalent to a job parameter --extra-py-files.

Default:

no extra python files and argument is not set

See:

https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html

Stability:

experimental

glue_version

(experimental) Glue version.

See:: https://docs.aws.amazon.com/glue/latest/dg/release-notes.html
Stability:: experimental

python_version

(experimental) The Python version to use.

Stability:: experimental

runtime

(experimental) Runtime.

It is required for Ray jobs.

Stability:: experimental

script

(experimental) The script that executes a job.

Stability:: experimental