Class PySparkStreamingJob

(experimental) Python Spark Streaming Jobs class.

Inheritance
object
Resource
JobBase
Job
SparkJob
PySparkStreamingJob
Implements
IConstruct
IDependable
Inherited Members
Namespace: Amazon.CDK.AWS.Glue.Alpha
Assembly: Amazon.CDK.AWS.Glue.Alpha.dll
Syntax (csharp)
public class PySparkStreamingJob : SparkJob, IJob, IResource, IConstruct, IDependable, IEnvironmentAware, IGrantable
Syntax (vb)
Public Class PySparkStreamingJob Inherits SparkJob Implements IJob, IResource, IConstruct, IDependable, IEnvironmentAware, IGrantable
Remarks

A Streaming job is similar to an ETL job, except that it performs ETL on data streams using the Apache Spark Structured Streaming framework. These jobs will default to use Python 3.9.

Similar to ETL jobs, streaming job supports Scala and Python languages. Similar to ETL, it supports G1 and G2 worker type and 2.0, 3.0 and 4.0 version. We’ll default to G2 worker and 4.0 version for streaming jobs which developers can override. We will enable —enable-metrics, —enable-spark-ui, —enable-continuous-cloudwatch-log.

Stability: Experimental

ExampleMetadata: infused

Examples
using Amazon.CDK;
             using Amazon.CDK.AWS.IAM;
             Stack stack;
             IRole role;
             Code script;

             new PySparkStreamingJob(stack, "ImportedJob", new PySparkStreamingJobProps { Role = role, Script = script });

Synopsis

Constructors

PySparkStreamingJob(Construct, string, IPySparkStreamingJobProps)

(experimental) PySparkStreamingJob constructor.

Properties

JobArn

(experimental) The ARN of the job.
JobName

(experimental) The name of the job.
PROPERTY_INJECTION_ID

(experimental) Uniquely identifies this class.

Constructors

PySparkStreamingJob(Construct, string, IPySparkStreamingJobProps)

(experimental) PySparkStreamingJob constructor.

public PySparkStreamingJob(Construct scope, string id, IPySparkStreamingJobProps props)
Parameters
scope Construct
id string
props IPySparkStreamingJobProps
Remarks

Stability: Experimental

Properties

JobArn

(experimental) The ARN of the job.

public override string JobArn { get; }
Property Value

string

Overrides
Remarks

Stability: Experimental

JobName

(experimental) The name of the job.

public override string JobName { get; }
Property Value

string

Overrides
Remarks

Stability: Experimental

PROPERTY_INJECTION_ID

(experimental) Uniquely identifies this class.

public static string PROPERTY_INJECTION_ID { get; }
Property Value

string

Remarks

Stability: Experimental

Implements

Constructs.IConstruct
Constructs.IDependable