S3JsonSource
Specifies a JSON data store stored in Amazon S3.
Contents
- Name
-
The name of the data store.
Type: String
Pattern:
([^\r\n])*Required: Yes
- Paths
-
A list of the Amazon S3 paths to read from.
Type: Array of strings
Pattern:
([\u0009\u000B\u000C\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF])*Required: Yes
- AdditionalOptions
-
Specifies additional connection options.
Type: S3DirectSourceAdditionalOptions object
Required: No
- CompressionType
-
Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are
"gzip"and"bzip").Type: String
Valid Values:
gzip | bzip2Required: No
- Exclusions
-
A string containing a JSON list of Unix-style glob patterns to exclude. For example, "[\"**.pdf\"]" excludes all PDF files.
Type: Array of strings
Pattern:
([\u0009\u000B\u000C\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF])*Required: No
- GroupFiles
-
Grouping files is turned on by default when the input contains more than 50,000 files. To turn on grouping with fewer than 50,000 files, set this parameter to "inPartition". To disable grouping when there are more than 50,000 files, set this parameter to
"none".Type: String
Pattern:
([\u0009\u000B\u000C\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF])*Required: No
- GroupSize
-
The target group size in bytes. The default is computed based on the input data size and the size of your cluster. When there are fewer than 50,000 input files,
"groupFiles"must be set to"inPartition"for this to take effect.Type: String
Pattern:
([\u0009\u000B\u000C\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF])*Required: No
- JsonPath
-
A JsonPath string defining the JSON data.
Type: String
Pattern:
([\u0009\u000B\u000C\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF])*Required: No
- MaxBand
-
This option controls the duration in milliseconds after which the s3 listing is likely to be consistent. Files with modification timestamps falling within the last maxBand milliseconds are tracked specially when using JobBookmarks to account for Amazon S3 eventual consistency. Most users don't need to set this option. The default is 900000 milliseconds, or 15 minutes.
Type: Integer
Valid Range: Minimum value of 0.
Required: No
- MaxFilesInBand
-
This option specifies the maximum number of files to save from the last maxBand seconds. If this number is exceeded, extra files are skipped and only processed in the next job run.
Type: Integer
Valid Range: Minimum value of 0.
Required: No
- Multiline
-
A Boolean value that specifies whether a single record can span multiple lines. This can occur when a field contains a quoted new-line character. You must set this option to True if any record spans multiple lines. The default value is
False, which allows for more aggressive file-splitting during parsing.Type: Boolean
Required: No
- OutputSchemas
-
Specifies the data schema for the S3 JSON source.
Type: Array of GlueSchema objects
Required: No
- Recurse
-
If set to true, recursively reads files in all subdirectories under the specified paths.
Type: Boolean
Required: No
See Also
For more information about using this API in one of the language-specific AWS SDKs, see the following:
