Class ScalaSparkStreamingJob
(experimental) Scala Streaming Jobs class.
Inherited Members
Namespace: Amazon.CDK.AWS.Glue.Alpha
Assembly: Amazon.CDK.AWS.Glue.Alpha.dll
Syntax (csharp)
public class ScalaSparkStreamingJob : SparkJob, IJob, IResource, IConstruct, IDependable, IEnvironmentAware, IGrantableSyntax (vb)
Public Class ScalaSparkStreamingJob Inherits SparkJob Implements IJob, IResource, IConstruct, IDependable, IEnvironmentAware, IGrantableRemarks
A Streaming job is similar to an ETL job, except that it performs ETL on data streams using the Apache Spark Structured Streaming framework. These jobs will default to use Python 3.9.
Similar to ETL jobs, streaming job supports Scala and Python languages. Similar to ETL, it supports G1 and G2 worker type and 2.0, 3.0 and 4.0 version. We’ll default to G2 worker and 4.0 version for streaming jobs which developers can override. We will enable —enable-metrics, —enable-spark-ui, —enable-continuous-cloudwatch-log.
Stability: Experimental
ExampleMetadata: fixture=_generated
Examples
// The code below shows an example of how to instantiate this type.
// The values are placeholders you should change.
using Amazon.CDK.AWS.Glue.Alpha;
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.Logs;
using Amazon.CDK.AWS.S3;
Bucket bucket;
Code code;
Connection connection;
LogGroup logGroup;
Role role;
SecurityConfiguration securityConfiguration;
var scalaSparkStreamingJob = new ScalaSparkStreamingJob(this, "MyScalaSparkStreamingJob", new ScalaSparkStreamingJobProps {
ClassName = "className",
Role = role,
Script = code,
// the properties below are optional
Connections = new [] { connection },
ContinuousLogging = new ContinuousLoggingProps {
Enabled = false,
// the properties below are optional
ConversionPattern = "conversionPattern",
LogGroup = logGroup,
LogStreamPrefix = "logStreamPrefix",
Quiet = false
},
DefaultArguments = new Dictionary<string, string> {
{ "defaultArgumentsKey", "defaultArguments" }
},
Description = "description",
EnableMetrics = false,
EnableObservabilityMetrics = false,
EnableProfilingMetrics = false,
ExtraFiles = new [] { code },
ExtraJars = new [] { code },
ExtraJarsFirst = false,
GlueVersion = GlueVersion.V0_9,
JobName = "jobName",
JobRunQueuingEnabled = false,
MaxConcurrentRuns = 123,
MaxRetries = 123,
NumberOfWorkers = 123,
SecurityConfiguration = securityConfiguration,
SparkUI = new SparkUIProps {
Bucket = bucket,
Prefix = "prefix"
},
Tags = new Dictionary<string, string> {
{ "tagsKey", "tags" }
},
Timeout = Duration.Minutes(30),
WorkerType = WorkerType.STANDARD
});Synopsis
Constructors
| ScalaSparkStreamingJob(Construct, string, IScalaSparkStreamingJobProps) | (experimental) ScalaSparkStreamingJob constructor. |
Properties
| JobArn | (experimental) The ARN of the job. |
| JobName | (experimental) The name of the job. |
| PROPERTY_INJECTION_ID | (experimental) Uniquely identifies this class. |
Constructors
ScalaSparkStreamingJob(Construct, string, IScalaSparkStreamingJobProps)
(experimental) ScalaSparkStreamingJob constructor.
public ScalaSparkStreamingJob(Construct scope, string id, IScalaSparkStreamingJobProps props)Parameters
- scope Construct
- id string
- props IScalaSparkStreamingJobProps
Remarks
Stability: Experimental
Properties
JobArn
(experimental) The ARN of the job.
public override string JobArn { get; }Property Value
Overrides
Remarks
Stability: Experimental
JobName
(experimental) The name of the job.
public override string JobName { get; }Property Value
Overrides
Remarks
Stability: Experimental
PROPERTY_INJECTION_ID
(experimental) Uniquely identifies this class.
public static string PROPERTY_INJECTION_ID { get; }Property Value
Remarks
Stability: Experimental