Interface IScalaSparkFlexEtlJobProps

(experimental) Flex Jobs class.

Inherited Members
Namespace: Amazon.CDK.AWS.Glue.Alpha
Assembly: Amazon.CDK.AWS.Glue.Alpha.dll
Syntax (csharp)
public interface IScalaSparkFlexEtlJobProps : IJobProperties
Syntax (vb)
Public Interface IScalaSparkFlexEtlJobProps
    Inherits IJobProperties
Remarks

Flex jobs supports Python and Scala language. The flexible execution class is appropriate for non-urgent jobs such as pre-production jobs, testing, and one-time data loads. Flexible job runs are supported for jobs using AWS Glue version 3.0 or later and G.1X or G.2X worker types but will default to the latest version of Glue (currently Glue 3.0.)

Similar to ETL, we’ll enable these features: —enable-metrics, —enable-spark-ui, —enable-continuous-cloudwatch-log

Stability: Experimental

ExampleMetadata: fixture=_generated

Examples
// The code below shows an example of how to instantiate this type.
// The values are placeholders you should change.
using Amazon.CDK.AWS.Glue.Alpha;
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.Logs;
using Amazon.CDK.AWS.S3;

Bucket bucket;
Code code;
Connection connection;
LogGroup logGroup;
Role role;
SecurityConfiguration securityConfiguration;

var scalaSparkFlexEtlJobProps = new ScalaSparkFlexEtlJobProps {
    ClassName = "className",
    Role = role,
    Script = code,

    // the properties below are optional
    Connections = new [] { connection },
    ContinuousLogging = new ContinuousLoggingProps {
        Enabled = false,

        // the properties below are optional
        ConversionPattern = "conversionPattern",
        LogGroup = logGroup,
        LogStreamPrefix = "logStreamPrefix",
        Quiet = false
    },
    DefaultArguments = new Dictionary<string, string> {
        { "defaultArgumentsKey", "defaultArguments" }
    },
    Description = "description",
    EnableProfilingMetrics = false,
    ExtraFiles = new [] { code },
    ExtraJars = new [] { code },
    ExtraJarsFirst = false,
    GlueVersion = GlueVersion.V0_9,
    JobName = "jobName",
    MaxConcurrentRuns = 123,
    MaxRetries = 123,
    NotifyDelayAfter = Duration.Minutes(30),
    NumberOfWorkers = 123,
    SecurityConfiguration = securityConfiguration,
    SparkUI = new SparkUIProps {
        Bucket = bucket,
        JobRunQueuingEnabled = false,
        Prefix = "prefix"
    },
    Tags = new Dictionary<string, string> {
        { "tagsKey", "tags" }
    },
    Timeout = Duration.Minutes(30),
    WorkerType = WorkerType.STANDARD
};

Synopsis

Properties

ClassName

(experimental) The fully qualified Scala class name that serves as the entry point for the job.
ExtraFiles

(experimental) Additional files, such as configuration files that AWS Glue copies to the working directory of your script before executing it.
ExtraJars

(experimental) Additional Java .jar files that AWS Glue adds to the Java classpath before executing your script. Only individual files are supported, directories are not supported.
ExtraJarsFirst

(experimental) Setting this value to true prioritizes the customer's extra JAR files in the classpath.
NotifyDelayAfter

(experimental) Specifies configuration properties of a notification (optional).
SparkUI

(experimental) Enables the Spark UI debugging and monitoring with the specified props.

Properties

ClassName

(experimental) The fully qualified Scala class name that serves as the entry point for the job.

string ClassName { get; }
Property Value

System.String

Remarks

Stability: Experimental

See: --class in https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html

ExtraFiles

(experimental) Additional files, such as configuration files that AWS Glue copies to the working directory of your script before executing it.

virtual Code[] ExtraFiles { get; }
Property Value

Code[]

Remarks

Only individual files are supported, directories are not supported.

Default: [] - no extra files are copied to the working directory

Stability: Experimental

See: --extra-files in https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html

ExtraJars

(experimental) Additional Java .jar files that AWS Glue adds to the Java classpath before executing your script. Only individual files are supported, directories are not supported.

virtual Code[] ExtraJars { get; }
Property Value

Code[]

Remarks

Default: [] - no extra jars are added to the classpath

Stability: Experimental

See: --extra-jars in https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html

ExtraJarsFirst

(experimental) Setting this value to true prioritizes the customer's extra JAR files in the classpath.

virtual Nullable<bool> ExtraJarsFirst { get; }
Property Value

System.Nullable<System.Boolean>

Remarks

Default: false - priority is not given to user-provided jars

Stability: Experimental

See: --user-jars-first in https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html

NotifyDelayAfter

(experimental) Specifies configuration properties of a notification (optional).

virtual Duration NotifyDelayAfter { get; }
Property Value

Duration

Remarks

After a job run starts, the number of minutes to wait before sending a job run delay notification.

Default: - undefined

Stability: Experimental

SparkUI

(experimental) Enables the Spark UI debugging and monitoring with the specified props.

virtual ISparkUIProps SparkUI { get; }
Property Value

ISparkUIProps

Remarks

Default: - Spark UI debugging and monitoring is disabled.

Stability: Experimental

See: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html