Class ScalaSparkFlexEtlJob
(experimental) Spark ETL Jobs class.
Inherited Members
Namespace: Amazon.CDK.AWS.Glue.Alpha
Assembly: Amazon.CDK.AWS.Glue.Alpha.dll
Syntax (csharp)
public class ScalaSparkFlexEtlJob : SparkJob, IJob, IResource, IConstruct, IDependable, IEnvironmentAware, IGrantableSyntax (vb)
Public Class ScalaSparkFlexEtlJob Inherits SparkJob Implements IJob, IResource, IConstruct, IDependable, IEnvironmentAware, IGrantableRemarks
ETL jobs support pySpark and Scala languages, for which there are separate but similar constructors. ETL jobs default to the G2 worker type, but you can override this default with other supported worker type values (G1, G2, G4 and G8). ETL jobs defaults to Glue version 4.0, which you can override to 3.0. The following ETL features are enabled by default: —enable-metrics, —enable-spark-ui, —enable-continuous-cloudwatch-log. You can find more details about version, worker type and other features in Glue's public documentation.
Stability: Experimental
ExampleMetadata: fixture=_generated
Examples
// The code below shows an example of how to instantiate this type.
// The values are placeholders you should change.
using Amazon.CDK.AWS.Glue.Alpha;
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.Logs;
using Amazon.CDK.AWS.S3;
Bucket bucket;
Code code;
Connection connection;
LogGroup logGroup;
Role role;
SecurityConfiguration securityConfiguration;
var scalaSparkFlexEtlJob = new ScalaSparkFlexEtlJob(this, "MyScalaSparkFlexEtlJob", new ScalaSparkFlexEtlJobProps {
ClassName = "className",
Role = role,
Script = code,
// the properties below are optional
Connections = new [] { connection },
ContinuousLogging = new ContinuousLoggingProps {
Enabled = false,
// the properties below are optional
ConversionPattern = "conversionPattern",
LogGroup = logGroup,
LogStreamPrefix = "logStreamPrefix",
Quiet = false
},
DefaultArguments = new Dictionary<string, string> {
{ "defaultArgumentsKey", "defaultArguments" }
},
Description = "description",
EnableMetrics = false,
EnableObservabilityMetrics = false,
EnableProfilingMetrics = false,
ExtraFiles = new [] { code },
ExtraJars = new [] { code },
ExtraJarsFirst = false,
GlueVersion = GlueVersion.V0_9,
JobName = "jobName",
MaxConcurrentRuns = 123,
MaxRetries = 123,
NotifyDelayAfter = Duration.Minutes(30),
NumberOfWorkers = 123,
SecurityConfiguration = securityConfiguration,
SparkUI = new SparkUIProps {
Bucket = bucket,
Prefix = "prefix"
},
Tags = new Dictionary<string, string> {
{ "tagsKey", "tags" }
},
Timeout = Duration.Minutes(30),
WorkerType = WorkerType.STANDARD
});Synopsis
Constructors
| ScalaSparkFlexEtlJob(Construct, string, IScalaSparkFlexEtlJobProps) | (experimental) ScalaSparkFlexEtlJob constructor. |
Properties
| JobArn | (experimental) The ARN of the job. |
| JobName | (experimental) The name of the job. |
| PROPERTY_INJECTION_ID | (experimental) Uniquely identifies this class. |
Constructors
ScalaSparkFlexEtlJob(Construct, string, IScalaSparkFlexEtlJobProps)
(experimental) ScalaSparkFlexEtlJob constructor.
public ScalaSparkFlexEtlJob(Construct scope, string id, IScalaSparkFlexEtlJobProps props)Parameters
- scope Construct
- id string
- props IScalaSparkFlexEtlJobProps
Remarks
Stability: Experimental
Properties
JobArn
(experimental) The ARN of the job.
public override string JobArn { get; }Property Value
Overrides
Remarks
Stability: Experimental
JobName
(experimental) The name of the job.
public override string JobName { get; }Property Value
Overrides
Remarks
Stability: Experimental
PROPERTY_INJECTION_ID
(experimental) Uniquely identifies this class.
public static string PROPERTY_INJECTION_ID { get; }Property Value
Remarks
Stability: Experimental