SparkSubmitJobDriver

class aws_cdk.aws_stepfunctions_tasks.SparkSubmitJobDriver(*, entry_point, entry_point_arguments=None, spark_submit_parameters=None)

Bases: object

The information about job driver for Spark submit.

Parameters:

entry_point (TaskInput) – The entry point of job application. Length Constraints: Minimum length of 1. Maximum length of 256.
entry_point_arguments (Optional[TaskInput]) – The arguments for a job application in a task input object containing an array of strings. Length Constraints: Minimum length of 1. Maximum length of 10280. Default: - No arguments defined
spark_submit_parameters (Optional[str]) – The Spark submit parameters that are used for job runs. Length Constraints: Minimum length of 1. Maximum length of 102400. Default: - No spark submit parameters

ExampleMetadata:

infused

Example:

tasks.EmrContainersStartJobRun(self, "EMR Containers Start Job Run",
    virtual_cluster=tasks.VirtualClusterInput.from_virtual_cluster_id("de92jdei2910fwedz"),
    release_label=tasks.ReleaseLabel.EMR_6_2_0,
    job_name="EMR-Containers-Job",
    job_driver=tasks.JobDriver(
        spark_submit_job_driver=tasks.SparkSubmitJobDriver(
            entry_point=sfn.TaskInput.from_text("local:///usr/lib/spark/examples/src/main/python/pi.py")
        )
    ),
    application_config=[tasks.ApplicationConfiguration(
        classification=tasks.Classification.SPARK_DEFAULTS,
        properties={
            "spark.executor.instances": "1",
            "spark.executor.memory": "512M"
        }
    )]
)

Attributes

entry_point

The entry point of job application.

Length Constraints: Minimum length of 1. Maximum length of 256.

entry_point_arguments

The arguments for a job application in a task input object containing an array of strings.

Length Constraints: Minimum length of 1. Maximum length of 10280.

Default:

No arguments defined

Type:

sfn.TaskInput which expects payload as an array of strings

spark_submit_parameters

The Spark submit parameters that are used for job runs.

Length Constraints: Minimum length of 1. Maximum length of 102400.

Default:

No spark submit parameters