Interface IScalaSparkFlexEtlJobProps

(experimental) Flex Jobs class.

Inherited Members
Namespace: Amazon.CDK.AWS.Glue.Alpha
Assembly: Amazon.CDK.AWS.Glue.Alpha.dll
Syntax (csharp)
public interface IScalaSparkFlexEtlJobProps : ISparkJobProps, IJobProps
Syntax (vb)
Public Interface IScalaSparkFlexEtlJobProps
    Inherits ISparkJobProps, IJobProps
Remarks

Flex jobs supports Python and Scala language. The flexible execution class is appropriate for non-urgent jobs such as pre-production jobs, testing, and one-time data loads. Flexible job runs are supported for jobs using AWS Glue version 3.0 or later and G.1X or G.2X worker types but will default to the latest version of Glue (currently Glue 3.0.)

Similar to ETL, we’ll enable these features: —enable-metrics, —enable-spark-ui, —enable-continuous-cloudwatch-log

Stability: Experimental

ExampleMetadata: fixture=_generated

Examples
// The code below shows an example of how to instantiate this type.
// The values are placeholders you should change.
using Amazon.CDK.AWS.Glue.Alpha;
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.Logs;
using Amazon.CDK.AWS.S3;

Bucket bucket;
Code code;
Connection connection;
LogGroup logGroup;
Role role;
SecurityConfiguration securityConfiguration;

var scalaSparkFlexEtlJobProps = new ScalaSparkFlexEtlJobProps {
    ClassName = "className",
    Role = role,
    Script = code,

    // the properties below are optional
    Connections = new [] { connection },
    ContinuousLogging = new ContinuousLoggingProps {
        Enabled = false,

        // the properties below are optional
        ConversionPattern = "conversionPattern",
        LogGroup = logGroup,
        LogStreamPrefix = "logStreamPrefix",
        Quiet = false
    },
    DefaultArguments = new Dictionary<string, string> {
        { "defaultArgumentsKey", "defaultArguments" }
    },
    Description = "description",
    EnableProfilingMetrics = false,
    ExtraFiles = new [] { code },
    ExtraJars = new [] { code },
    ExtraJarsFirst = false,
    GlueVersion = GlueVersion.V0_9,
    JobName = "jobName",
    MaxConcurrentRuns = 123,
    MaxRetries = 123,
    NotifyDelayAfter = Duration.Minutes(30),
    NumberOfWorkers = 123,
    SecurityConfiguration = securityConfiguration,
    SparkUI = new SparkUIProps {
        Bucket = bucket,
        Prefix = "prefix"
    },
    Tags = new Dictionary<string, string> {
        { "tagsKey", "tags" }
    },
    Timeout = Duration.Minutes(30),
    WorkerType = WorkerType.STANDARD
};

Synopsis

Properties

ClassName

(experimental) The fully qualified Scala class name that serves as the entry point for the job.
ExtraFiles

(experimental) Additional files, such as configuration files that AWS Glue copies to the working directory of your script before executing it.
ExtraJars

(experimental) Extra Jars S3 URL (optional) S3 URL where additional jar dependencies are located.
ExtraJarsFirst

(experimental) Setting this value to true prioritizes the customer's extra JAR files in the classpath.
NotifyDelayAfter

(experimental) Specifies configuration properties of a notification (optional).

Properties

ClassName

(experimental) The fully qualified Scala class name that serves as the entry point for the job.

string ClassName { get; }
Property Value

System.String

Remarks

Stability: Experimental

See: --class in https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html

ExtraFiles

(experimental) Additional files, such as configuration files that AWS Glue copies to the working directory of your script before executing it.

virtual Code[] ExtraFiles { get; }
Property Value

Code[]

Remarks

Default: - no extra files specified.

Stability: Experimental

See: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html

ExtraJars

(experimental) Extra Jars S3 URL (optional) S3 URL where additional jar dependencies are located.

virtual Code[] ExtraJars { get; }
Property Value

Code[]

Remarks

Default: - no extra jar files

Stability: Experimental

ExtraJarsFirst

(experimental) Setting this value to true prioritizes the customer's extra JAR files in the classpath.

virtual Nullable<bool> ExtraJarsFirst { get; }
Property Value

System.Nullable<System.Boolean>

Remarks

Default: false - priority is not given to user-provided jars

Stability: Experimental

See: --user-jars-first in https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html

NotifyDelayAfter

(experimental) Specifies configuration properties of a notification (optional).

virtual Duration NotifyDelayAfter { get; }
Property Value

Duration

Remarks

After a job run starts, the number of minutes to wait before sending a job run delay notification.

Default: - undefined

Stability: Experimental