Class SparkJob

(experimental) Base class for different types of Spark Jobs.

Inheritance

object

Resource

SparkJob

ScalaSparkStreamingJob

Implements

IJob

IResource

IConstruct

IDependable

IGrantable

Inherited Members

Job.FromJobAttributes(Construct, string, IJobAttributes)

Job.CheckNoReservedArgs(IDictionary<string, string>)

Job.CodeS3ObjectUrl(Code)

Job.SetupContinuousLogging(IRole, IContinuousLoggingProps)

JobBase.BuildJobArn(Construct, string)

JobBase.Metric(string, MetricType, IMetricOptions)

JobBase.MetricFailure(IMetricOptions)

JobBase.MetricSuccess(IMetricOptions)

JobBase.MetricTimeout(IMetricOptions)

JobBase.OnEvent(string, IOnEventOptions)

JobBase.OnFailure(string, IOnEventOptions)

JobBase.OnStateChange(string, JobState, IOnEventOptions)

JobBase.OnSuccess(string, IOnEventOptions)

JobBase.OnTimeout(string, IOnEventOptions)

JobBase.JobArn

JobBase.JobName

Resource.IsOwnedResource(IConstruct)

Resource.IsResource(IConstruct)

Resource.ApplyRemovalPolicy(RemovalPolicy)

Resource.GeneratePhysicalName()

Resource.GetResourceArnAttribute(string, IArnComponents)

Resource.GetResourceNameAttribute(string)

Resource.Env

Resource.PhysicalName

Resource.Stack

Namespace: Amazon.CDK.AWS.Glue.Alpha

Assembly: Amazon.CDK.AWS.Glue.Alpha.dll

Syntax (csharp)

public abstract class SparkJob : Job, IJob, IResource, IConstruct, IDependable, IGrantable

Syntax (vb)

Public MustInherit Class SparkJob Inherits Job Implements IJob, IResource, IConstruct, IDependable, IGrantable

Remarks

Stability: Experimental

ExampleMetadata: fixture=_generated

Examples

// The code below shows an example of how to instantiate this type.
             // The values are placeholders you should change.
             using Amazon.CDK.AWS.Glue.Alpha;
             using Amazon.CDK.AWS.IAM;

             Role role;

             var sparkJob = SparkJob.FromJobAttributes(this, "MySparkJob", new JobAttributes {
                 JobName = "jobName",

                 // the properties below are optional
                 Role = role
             });

Synopsis

Constructors

SparkJob(Construct, string, ISparkJobProps)

(experimental) Base class for different types of Spark Jobs.

Properties

GrantPrincipal	(experimental) The principal to grant permissions to.
Role	(experimental) The IAM role Glue assumes to run this job.
SparkUILoggingLocation	(experimental) The Spark UI logs location if Spark UI monitoring and debugging is enabled.

Methods

NonExecutableCommonArguments(ISparkJobProps)	(experimental) Base class for different types of Spark Jobs.
SetupExtraCodeArguments(IDictionary<string, string>, ISparkExtraCodeProps)	(experimental) Set the arguments for extra {@link Code}-related properties.

Constructors

SparkJob(Construct, string, ISparkJobProps)

(experimental) Base class for different types of Spark Jobs.

protected SparkJob(Construct scope, string id, ISparkJobProps props)

Parameters

scope Construct
id string
props ISparkJobProps

Remarks

Stability: Experimental

Properties

GrantPrincipal

(experimental) The principal to grant permissions to.

public override IPrincipal GrantPrincipal { get; }

Property Value

IPrincipal

Overrides

JobBase.GrantPrincipal

Remarks

Stability: Experimental

Role

(experimental) The IAM role Glue assumes to run this job.

public override IRole Role { get; }

Property Value

IRole

Overrides

Job.Role

Remarks

Stability: Experimental

SparkUILoggingLocation

(experimental) The Spark UI logs location if Spark UI monitoring and debugging is enabled.

public virtual ISparkUILoggingLocation? SparkUILoggingLocation { get; }

Property Value

ISparkUILoggingLocation

Remarks

Stability: Experimental

See: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html

Methods

NonExecutableCommonArguments(ISparkJobProps)

(experimental) Base class for different types of Spark Jobs.

protected virtual IDictionary<string, string> NonExecutableCommonArguments(ISparkJobProps props)

Parameters

props ISparkJobProps

Returns

IDictionary<string, string>

Remarks

Stability: Experimental

SetupExtraCodeArguments(IDictionary<string, string>, ISparkExtraCodeProps)

(experimental) Set the arguments for extra {@link Code}-related properties.

protected virtual void SetupExtraCodeArguments(IDictionary<string, string> args, ISparkExtraCodeProps props)

Parameters

args IDictionary<string, string>
props ISparkExtraCodeProps

Remarks

Stability: Experimental

Implements

IJob

IResource

Constructs.IConstruct

Constructs.IDependable

IGrantable