Class PySparkEtlJob

(experimental) PySpark ETL Jobs class.

Inheritance
object
Resource
JobBase
Job
SparkJob
PySparkEtlJob
Implements
IConstruct
IDependable
Inherited Members
Namespace: Amazon.CDK.AWS.Glue.Alpha
Assembly: Amazon.CDK.AWS.Glue.Alpha.dll
Syntax (csharp)
public class PySparkEtlJob : SparkJob, IJob, IResource, IConstruct, IDependable, IEnvironmentAware, IGrantable
Syntax (vb)
Public Class PySparkEtlJob Inherits SparkJob Implements IJob, IResource, IConstruct, IDependable, IEnvironmentAware, IGrantable
Remarks

ETL jobs support pySpark and Scala languages, for which there are separate but similar constructors. ETL jobs default to the G2 worker type, but you can override this default with other supported worker type values (G1, G2, G4 and G8). ETL jobs defaults to Glue version 4.0, which you can override to 3.0. The following ETL features are enabled by default: —enable-metrics, —enable-spark-ui, —enable-continuous-cloudwatch-log. You can find more details about version, worker type and other features in Glue's public documentation.

Stability: Experimental

ExampleMetadata: infused

Examples
using Amazon.CDK;
             using Amazon.CDK.AWS.IAM;
             Stack stack;
             IRole role;
             Code script;


             // Disable both metrics for cost optimization
             // Disable both metrics for cost optimization
             new PySparkEtlJob(stack, "CostOptimizedJob", new PySparkEtlJobProps {
                 Role = role,
                 Script = script,
                 EnableMetrics = false,
                 EnableObservabilityMetrics = false
             });

             // Selective control - keep observability, disable profiling
             // Selective control - keep observability, disable profiling
             new PySparkEtlJob(stack, "SelectiveJob", new PySparkEtlJobProps {
                 Role = role,
                 Script = script,
                 EnableMetrics = false
             });

Synopsis

Constructors

PySparkEtlJob(Construct, string, IPySparkEtlJobProps)

(experimental) PySparkEtlJob constructor.

Properties

JobArn

(experimental) The ARN of the job.
JobName

(experimental) The name of the job.
PROPERTY_INJECTION_ID

(experimental) Uniquely identifies this class.

Constructors

PySparkEtlJob(Construct, string, IPySparkEtlJobProps)

(experimental) PySparkEtlJob constructor.

public PySparkEtlJob(Construct scope, string id, IPySparkEtlJobProps props)
Parameters
scope Construct
id string
props IPySparkEtlJobProps
Remarks

Stability: Experimental

Properties

JobArn

(experimental) The ARN of the job.

public override string JobArn { get; }
Property Value

string

Overrides
Remarks

Stability: Experimental

JobName

(experimental) The name of the job.

public override string JobName { get; }
Property Value

string

Overrides
Remarks

Stability: Experimental

PROPERTY_INJECTION_ID

(experimental) Uniquely identifies this class.

public static string PROPERTY_INJECTION_ID { get; }
Property Value

string

Remarks

Stability: Experimental

Implements

Constructs.IConstruct
Constructs.IDependable