Class: Aws::Glue::Types::S3ParquetSource

Inherits:
Struct
  • Object
show all
Defined in:
gems/aws-sdk-glue/lib/aws-sdk-glue/types.rb

Overview

Note:

When making an API call, you may pass S3ParquetSource data as a hash:

{
  name: "NodeName", # required
  paths: ["EnclosedInStringProperty"], # required
  compression_type: "snappy", # accepts snappy, lzo, gzip, uncompressed, none
  exclusions: ["EnclosedInStringProperty"],
  group_size: "EnclosedInStringProperty",
  group_files: "EnclosedInStringProperty",
  recurse: false,
  max_band: 1,
  max_files_in_band: 1,
  additional_options: {
    bounded_size: 1,
    bounded_files: 1,
    enable_sample_path: false,
    sample_path: "EnclosedInStringProperty",
  },
  output_schemas: [
    {
      columns: [
        {
          name: "GlueStudioColumnNameString", # required
          type: "ColumnTypeString",
        },
      ],
    },
  ],
}

Specifies an Apache Parquet data store stored in Amazon S3.

Constant Summary collapse

SENSITIVE =
[]

Instance Attribute Summary collapse

Instance Attribute Details

#additional_optionsTypes::S3DirectSourceAdditionalOptions

Specifies additional connection options.



20793
20794
20795
20796
20797
20798
20799
20800
20801
20802
20803
20804
20805
20806
20807
# File 'gems/aws-sdk-glue/lib/aws-sdk-glue/types.rb', line 20793

class S3ParquetSource < Struct.new(
  :name,
  :paths,
  :compression_type,
  :exclusions,
  :group_size,
  :group_files,
  :recurse,
  :max_band,
  :max_files_in_band,
  :additional_options,
  :output_schemas)
  SENSITIVE = []
  include Aws::Structure
end

#compression_typeString

Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are "gzip" and "bzip").

Returns:

  • (String)


20793
20794
20795
20796
20797
20798
20799
20800
20801
20802
20803
20804
20805
20806
20807
# File 'gems/aws-sdk-glue/lib/aws-sdk-glue/types.rb', line 20793

class S3ParquetSource < Struct.new(
  :name,
  :paths,
  :compression_type,
  :exclusions,
  :group_size,
  :group_files,
  :recurse,
  :max_band,
  :max_files_in_band,
  :additional_options,
  :output_schemas)
  SENSITIVE = []
  include Aws::Structure
end

#exclusionsArray<String>

A string containing a JSON list of Unix-style glob patterns to exclude. For example, "[\"**.pdf\"]" excludes all PDF files.

Returns:

  • (Array<String>)


20793
20794
20795
20796
20797
20798
20799
20800
20801
20802
20803
20804
20805
20806
20807
# File 'gems/aws-sdk-glue/lib/aws-sdk-glue/types.rb', line 20793

class S3ParquetSource < Struct.new(
  :name,
  :paths,
  :compression_type,
  :exclusions,
  :group_size,
  :group_files,
  :recurse,
  :max_band,
  :max_files_in_band,
  :additional_options,
  :output_schemas)
  SENSITIVE = []
  include Aws::Structure
end

#group_filesString

Grouping files is turned on by default when the input contains more than 50,000 files. To turn on grouping with fewer than 50,000 files, set this parameter to "inPartition". To disable grouping when there are more than 50,000 files, set this parameter to "none".

Returns:

  • (String)


20793
20794
20795
20796
20797
20798
20799
20800
20801
20802
20803
20804
20805
20806
20807
# File 'gems/aws-sdk-glue/lib/aws-sdk-glue/types.rb', line 20793

class S3ParquetSource < Struct.new(
  :name,
  :paths,
  :compression_type,
  :exclusions,
  :group_size,
  :group_files,
  :recurse,
  :max_band,
  :max_files_in_band,
  :additional_options,
  :output_schemas)
  SENSITIVE = []
  include Aws::Structure
end

#group_sizeString

The target group size in bytes. The default is computed based on the input data size and the size of your cluster. When there are fewer than 50,000 input files, "groupFiles" must be set to "inPartition" for this to take effect.

Returns:

  • (String)


20793
20794
20795
20796
20797
20798
20799
20800
20801
20802
20803
20804
20805
20806
20807
# File 'gems/aws-sdk-glue/lib/aws-sdk-glue/types.rb', line 20793

class S3ParquetSource < Struct.new(
  :name,
  :paths,
  :compression_type,
  :exclusions,
  :group_size,
  :group_files,
  :recurse,
  :max_band,
  :max_files_in_band,
  :additional_options,
  :output_schemas)
  SENSITIVE = []
  include Aws::Structure
end

#max_bandInteger

This option controls the duration in milliseconds after which the s3 listing is likely to be consistent. Files with modification timestamps falling within the last maxBand milliseconds are tracked specially when using JobBookmarks to account for Amazon S3 eventual consistency. Most users don't need to set this option. The default is 900000 milliseconds, or 15 minutes.

Returns:

  • (Integer)


20793
20794
20795
20796
20797
20798
20799
20800
20801
20802
20803
20804
20805
20806
20807
# File 'gems/aws-sdk-glue/lib/aws-sdk-glue/types.rb', line 20793

class S3ParquetSource < Struct.new(
  :name,
  :paths,
  :compression_type,
  :exclusions,
  :group_size,
  :group_files,
  :recurse,
  :max_band,
  :max_files_in_band,
  :additional_options,
  :output_schemas)
  SENSITIVE = []
  include Aws::Structure
end

#max_files_in_bandInteger

This option specifies the maximum number of files to save from the last maxBand seconds. If this number is exceeded, extra files are skipped and only processed in the next job run.

Returns:

  • (Integer)


20793
20794
20795
20796
20797
20798
20799
20800
20801
20802
20803
20804
20805
20806
20807
# File 'gems/aws-sdk-glue/lib/aws-sdk-glue/types.rb', line 20793

class S3ParquetSource < Struct.new(
  :name,
  :paths,
  :compression_type,
  :exclusions,
  :group_size,
  :group_files,
  :recurse,
  :max_band,
  :max_files_in_band,
  :additional_options,
  :output_schemas)
  SENSITIVE = []
  include Aws::Structure
end

#nameString

The name of the data store.

Returns:

  • (String)


20793
20794
20795
20796
20797
20798
20799
20800
20801
20802
20803
20804
20805
20806
20807
# File 'gems/aws-sdk-glue/lib/aws-sdk-glue/types.rb', line 20793

class S3ParquetSource < Struct.new(
  :name,
  :paths,
  :compression_type,
  :exclusions,
  :group_size,
  :group_files,
  :recurse,
  :max_band,
  :max_files_in_band,
  :additional_options,
  :output_schemas)
  SENSITIVE = []
  include Aws::Structure
end

#output_schemasArray<Types::GlueSchema>

Specifies the data schema for the S3 Parquet source.

Returns:



20793
20794
20795
20796
20797
20798
20799
20800
20801
20802
20803
20804
20805
20806
20807
# File 'gems/aws-sdk-glue/lib/aws-sdk-glue/types.rb', line 20793

class S3ParquetSource < Struct.new(
  :name,
  :paths,
  :compression_type,
  :exclusions,
  :group_size,
  :group_files,
  :recurse,
  :max_band,
  :max_files_in_band,
  :additional_options,
  :output_schemas)
  SENSITIVE = []
  include Aws::Structure
end

#pathsArray<String>

A list of the Amazon S3 paths to read from.

Returns:

  • (Array<String>)


20793
20794
20795
20796
20797
20798
20799
20800
20801
20802
20803
20804
20805
20806
20807
# File 'gems/aws-sdk-glue/lib/aws-sdk-glue/types.rb', line 20793

class S3ParquetSource < Struct.new(
  :name,
  :paths,
  :compression_type,
  :exclusions,
  :group_size,
  :group_files,
  :recurse,
  :max_band,
  :max_files_in_band,
  :additional_options,
  :output_schemas)
  SENSITIVE = []
  include Aws::Structure
end

#recurseBoolean

If set to true, recursively reads files in all subdirectories under the specified paths.

Returns:

  • (Boolean)


20793
20794
20795
20796
20797
20798
20799
20800
20801
20802
20803
20804
20805
20806
20807
# File 'gems/aws-sdk-glue/lib/aws-sdk-glue/types.rb', line 20793

class S3ParquetSource < Struct.new(
  :name,
  :paths,
  :compression_type,
  :exclusions,
  :group_size,
  :group_files,
  :recurse,
  :max_band,
  :max_files_in_band,
  :additional_options,
  :output_schemas)
  SENSITIVE = []
  include Aws::Structure
end