PythonShellJobProps

class aws_cdk.aws_glue_alpha.PythonShellJobProps(*, role, script, connections=None, continuous_logging=None, default_arguments=None, description=None, glue_version=None, job_name=None, max_concurrent_runs=None, max_retries=None, security_configuration=None, tags=None, timeout=None, extra_python_files=None, job_run_queuing_enabled=None, library_set=None, max_capacity=None, python_version=None)

Bases: JobProps

(experimental) Properties for creating a Python Shell job.

Parameters:

  • role (IRole) – (experimental) IAM Role (required) IAM Role to use for Glue job execution Must be specified by the developer because the L2 doesn’t have visibility into the actions the script(s) takes during the job execution The role must trust the Glue service principal (glue.amazonaws.com) and be granted sufficient permissions.

  • script (Code) – (experimental) Script Code Location (required) Script to run when the Glue job executes. Can be uploaded from the local directory structure using fromAsset or referenced via S3 location using fromBucket

  • connections (Optional[Sequence[IConnection]]) – (experimental) Connections (optional) List of connections to use for this Glue job Connections are used to connect to other AWS Service or resources within a VPC. Default: [] - no connections are added to the job

  • continuous_logging (Union[ContinuousLoggingProps, Dict[str, Any], None]) – (experimental) Enables continuous logging with the specified props. Default: - continuous logging is enabled.

  • default_arguments (Optional[Mapping[str, str]]) – (experimental) Default Arguments (optional) The default arguments for every run of this Glue job, specified as name-value pairs. This map is the escape hatch for Glue job arguments that this construct does not model. It MUST NOT be used to set arguments that already have a dedicated prop — configure those through the corresponding prop instead (continuousLogging, enableMetrics, enableObservabilityMetrics, sparkUI, className, extraJars, extraJarsFirst, extraPythonFiles, extraFiles). Passing a construct-managed argument (e.g. --enable-continuous-cloudwatch-log, --enable-metrics, --enable-spark-ui, --job-language) or a Glue-reserved argument (--debug, --mode, --JOB_NAME, --endpoint) here throws at synthesis time, so there is exactly one way to express each intent. Also note that these are emitted verbatim into the CloudFormation template, so avoid placing secrets here in plaintext. Pass secrets to the job at runtime through AWS Secrets Manager instead. A synthesis-time warning is emitted when an argument key looks like a credential and holds a plaintext literal. Default: - no arguments

  • description (Optional[str]) – (experimental) Description (optional) Developer-specified description of the Glue job. Default: - no value

  • glue_version (Optional[GlueVersion]) – (experimental) Glue Version The version of Glue to use to execute this job. Default: - determined by the job type: 4.0 for ETL and Streaming, 5.0 for Flex, 3.0 for Python Shell

  • job_name (Optional[str]) – (experimental) Name of the Glue job (optional) Developer-specified name of the Glue job. Default: - a name is automatically generated

  • max_concurrent_runs (Union[int, float, None]) – (experimental) Max Concurrent Runs (optional) The maximum number of runs this Glue job can concurrently run. An error is returned when this threshold is reached. The maximum value you can specify is controlled by a service limit. Default: 1

  • max_retries (Union[int, float, None]) – (experimental) Max Retries (optional) Maximum number of retry attempts Glue performs if the job fails. Default: 0

  • security_configuration (Optional[ISecurityConfiguration]) – (experimental) Security Configuration (optional) Defines the encryption options for the Glue job. Default: - no security configuration.

  • tags (Optional[Mapping[str, str]]) – (experimental) Tags (optional) A list of key:value pairs of tags to apply to this Glue job resources. Default: {} - no tags

  • timeout (Optional[Duration]) – (experimental) Timeout (optional) The maximum time that a job run can consume resources before it is terminated and enters TIMEOUT status. Specified in minutes. Default: 2880 (2 days for non-streaming)

  • extra_python_files (Optional[Sequence[Code]]) – (experimental) Additional Python files that AWS Glue adds to the Python path before executing your script. Only individual files are supported, directories are not supported. Equivalent to the --extra-py-files job argument. Default: - no extra Python files

  • job_run_queuing_enabled (Optional[bool]) – (experimental) Specifies whether job run queuing is enabled for the job runs for this job. A value of true means job run queuing is enabled for the job runs. If false or not populated, the job runs will not be considered for queueing. If this field does not match the value set in the job run, then the value from the job run field will be used. This property must be set to false for flex jobs. If this property is enabled, maxRetries must be set to zero. Default: false

  • library_set (Optional[LibrarySet]) – (experimental) The set of pre-installed Python libraries to make available to the job. Only applies to jobs running Python 3.9. Set to LibrarySet.NONE when your libraries are custom or conflict with the pre-installed ones. Default: LibrarySet.ANALYTICS when running Python 3.9, otherwise no library set is configured

  • max_capacity (Optional[MaxCapacity]) – (experimental) The total number of DPU to assign to the Python Job. Default: 0.0625

  • python_version (Optional[PythonVersion]) – (experimental) The version of Python to use to execute this job. Python shell jobs only support PythonVersion.THREE_NINE. The older PythonVersion.TWO (Python 2.7) and PythonVersion.THREE (Python 3.6) runtimes have been retired by AWS Glue and are no longer available for Python shell jobs. Default: PythonVersion.THREE_NINE

Stability:

experimental

ExampleMetadata:

infused

Example:

import aws_cdk as cdk
import aws_cdk.aws_iam as iam
# stack: cdk.Stack
# role: iam.IRole
# script: glue.Code

glue.PythonShellJob(stack, "ImportedJob", role=role, script=script)

Attributes

connections

(experimental) Connections (optional) List of connections to use for this Glue job Connections are used to connect to other AWS Service or resources within a VPC.

Default:

[] - no connections are added to the job

Stability:

experimental

continuous_logging

(experimental) Enables continuous logging with the specified props.

Default:

  • continuous logging is enabled.

See:

https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html

Stability:

experimental

default_arguments

(experimental) Default Arguments (optional) The default arguments for every run of this Glue job, specified as name-value pairs.

This map is the escape hatch for Glue job arguments that this construct does not model. It MUST NOT be used to set arguments that already have a dedicated prop — configure those through the corresponding prop instead (continuousLogging, enableMetrics, enableObservabilityMetrics, sparkUI, className, extraJars, extraJarsFirst, extraPythonFiles, extraFiles). Passing a construct-managed argument (e.g. --enable-continuous-cloudwatch-log, --enable-metrics, --enable-spark-ui, --job-language) or a Glue-reserved argument (--debug, --mode, --JOB_NAME, --endpoint) here throws at synthesis time, so there is exactly one way to express each intent.

Also note that these are emitted verbatim into the CloudFormation template, so avoid placing secrets here in plaintext. Pass secrets to the job at runtime through AWS Secrets Manager instead. A synthesis-time warning is emitted when an argument key looks like a credential and holds a plaintext literal.

Default:

  • no arguments

See:

https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html for a list of reserved parameters :stability: experimental

description

(experimental) Description (optional) Developer-specified description of the Glue job.

Default:

  • no value

Stability:

experimental

extra_python_files

(experimental) Additional Python files that AWS Glue adds to the Python path before executing your script.

Only individual files are supported, directories are not supported. Equivalent to the --extra-py-files job argument.

Default:

  • no extra Python files

See:

https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html

Stability:

experimental

glue_version

(experimental) Glue Version The version of Glue to use to execute this job.

Default:

  • determined by the job type: 4.0 for ETL and Streaming, 5.0 for Flex, 3.0 for Python Shell

Stability:

experimental

job_name

(experimental) Name of the Glue job (optional) Developer-specified name of the Glue job.

Default:

  • a name is automatically generated

Stability:

experimental

job_run_queuing_enabled

(experimental) Specifies whether job run queuing is enabled for the job runs for this job.

A value of true means job run queuing is enabled for the job runs. If false or not populated, the job runs will not be considered for queueing. If this field does not match the value set in the job run, then the value from the job run field will be used. This property must be set to false for flex jobs. If this property is enabled, maxRetries must be set to zero.

Default:

false

Stability:

experimental

library_set

(experimental) The set of pre-installed Python libraries to make available to the job.

Only applies to jobs running Python 3.9. Set to LibrarySet.NONE when your libraries are custom or conflict with the pre-installed ones.

Default:

LibrarySet.ANALYTICS when running Python 3.9, otherwise no library set is configured

See:

https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html#python-shell-supported-library

Stability:

experimental

max_capacity

(experimental) The total number of DPU to assign to the Python Job.

Default:

0.0625

Stability:

experimental

max_concurrent_runs

(experimental) Max Concurrent Runs (optional) The maximum number of runs this Glue job can concurrently run.

An error is returned when this threshold is reached. The maximum value you can specify is controlled by a service limit.

Default:

1

Stability:

experimental

max_retries

(experimental) Max Retries (optional) Maximum number of retry attempts Glue performs if the job fails.

Default:

0

Stability:

experimental

python_version

(experimental) The version of Python to use to execute this job.

Python shell jobs only support PythonVersion.THREE_NINE. The older PythonVersion.TWO (Python 2.7) and PythonVersion.THREE (Python 3.6) runtimes have been retired by AWS Glue and are no longer available for Python shell jobs.

Default:

PythonVersion.THREE_NINE

Stability:

experimental

role

(experimental) IAM Role (required) IAM Role to use for Glue job execution Must be specified by the developer because the L2 doesn’t have visibility into the actions the script(s) takes during the job execution The role must trust the Glue service principal (glue.amazonaws.com) and be granted sufficient permissions.

See:

https://docs.aws.amazon.com/glue/latest/dg/getting-started-access.html

Stability:

experimental

script

(experimental) Script Code Location (required) Script to run when the Glue job executes.

Can be uploaded from the local directory structure using fromAsset or referenced via S3 location using fromBucket

Stability:

experimental

security_configuration

(experimental) Security Configuration (optional) Defines the encryption options for the Glue job.

Default:

  • no security configuration.

Stability:

experimental

tags

value pairs of tags to apply to this Glue job resources.

Default:

{} - no tags

Stability:

experimental

Type:

(experimental) Tags (optional) A list of key

timeout

(experimental) Timeout (optional) The maximum time that a job run can consume resources before it is terminated and enters TIMEOUT status.

Specified in minutes.

Default:

2880 (2 days for non-streaming)

Stability:

experimental