KinesisStreamingSourceOptions
Additional options for the Amazon Kinesis streaming data source.
Contents
- AddIdleTimeBetweenReads
-
Adds a time delay between two consecutive getRecords operations. The default value is
"False"
. This option is only configurable for Glue version 2.0 and above.Type: Boolean
Required: No
- AddRecordTimestamp
-
When this option is set to 'true', the data output will contain an additional column named "__src_timestamp" that indicates the time when the corresponding record received by the stream. The default value is 'false'. This option is supported in AWS Glue version 4.0 or later.
Type: String
Pattern:
([\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF]|[^\S\r\n"'])*
Required: No
- AvoidEmptyBatches
-
Avoids creating an empty microbatch job by checking for unread data in the Kinesis data stream before the batch is started. The default value is
"False"
.Type: Boolean
Required: No
- Classification
-
An optional classification.
Type: String
Pattern:
([\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF]|[^\S\r\n"'])*
Required: No
- Delimiter
-
Specifies the delimiter character.
Type: String
Pattern:
([\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF]|[^\S\r\n"'])*
Required: No
- DescribeShardInterval
-
The minimum time interval between two ListShards API calls for your script to consider resharding. The default value is
1s
.Type: Long
Valid Range: Minimum value of 0.
Required: No
- EmitConsumerLagMetrics
-
When this option is set to 'true', for each batch, it will emit the metrics for the duration between the oldest record received by the stream and the time it arrives in AWS Glue to CloudWatch. The metric's name is "glue.driver.streaming.maxConsumerLagInMs". The default value is 'false'. This option is supported in AWS Glue version 4.0 or later.
Type: String
Pattern:
([\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF]|[^\S\r\n"'])*
Required: No
- EndpointUrl
-
The URL of the Kinesis endpoint.
Type: String
Pattern:
([\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF]|[^\S\r\n"'])*
Required: No
- IdleTimeBetweenReadsInMs
-
The minimum time delay between two consecutive getRecords operations, specified in ms. The default value is
1000
. This option is only configurable for Glue version 2.0 and above.Type: Long
Valid Range: Minimum value of 0.
Required: No
- MaxFetchRecordsPerShard
-
The maximum number of records to fetch per shard in the Kinesis data stream. The default value is
100000
.Type: Long
Valid Range: Minimum value of 0.
Required: No
- MaxFetchTimeInMs
-
The maximum time spent in the job executor to fetch a record from the Kinesis data stream per shard, specified in milliseconds (ms). The default value is
1000
.Type: Long
Valid Range: Minimum value of 0.
Required: No
- MaxRecordPerRead
-
The maximum number of records to fetch from the Kinesis data stream in each getRecords operation. The default value is
10000
.Type: Long
Valid Range: Minimum value of 0.
Required: No
- MaxRetryIntervalMs
-
The maximum cool-off time period (specified in ms) between two retries of a Kinesis Data Streams API call. The default value is
10000
.Type: Long
Valid Range: Minimum value of 0.
Required: No
- NumRetries
-
The maximum number of retries for Kinesis Data Streams API requests. The default value is
3
.Type: Integer
Valid Range: Minimum value of 0.
Required: No
- RetryIntervalMs
-
The cool-off time period (specified in ms) before retrying the Kinesis Data Streams API call. The default value is
1000
.Type: Long
Valid Range: Minimum value of 0.
Required: No
- RoleArn
-
The Amazon Resource Name (ARN) of the role to assume using AWS Security Token Service (AWS STS). This role must have permissions for describe or read record operations for the Kinesis data stream. You must use this parameter when accessing a data stream in a different account. Used in conjunction with
"awsSTSSessionName"
.Type: String
Pattern:
([\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF]|[^\S\r\n"'])*
Required: No
- RoleSessionName
-
An identifier for the session assuming the role using AWS STS. You must use this parameter when accessing a data stream in a different account. Used in conjunction with
"awsSTSRoleARN"
.Type: String
Pattern:
([\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF]|[^\S\r\n"'])*
Required: No
- StartingPosition
-
The starting position in the Kinesis data stream to read data from. The possible values are
"latest"
,"trim_horizon"
, or"earliest"
. The default value is"latest"
.Type: String
Valid Values:
latest | trim_horizon | earliest
Required: No
- StreamArn
-
The Amazon Resource Name (ARN) of the Kinesis data stream.
Type: String
Pattern:
([\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF]|[^\S\r\n"'])*
Required: No
- StreamName
-
The name of the Kinesis data stream.
Type: String
Pattern:
([\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF]|[^\S\r\n"'])*
Required: No
See Also
For more information about using this API in one of the language-specific AWS SDKs, see the following: