Common data types - AWS Glue

Common data types

The Common data types describes miscellaneous common data types in AWS Glue.

Tag structure

The Tag object represents a label that you can assign to an AWS resource. Each tag consists of a key and an optional value, both of which you define.

For more information about tags, and controlling access to resources in AWS Glue, see AWS Tags in AWS Glue and Specifying AWS Glue Resource ARNs in the developer guide.

Fields
  • key – UTF-8 string, not less than 1 or more than 128 bytes long.

    The tag key. The key is required when you create a tag on an object. The key is case-sensitive, and must not contain the prefix aws.

  • value – UTF-8 string, not more than 256 bytes long.

    The tag value. The value is optional when you create a tag on an object. The value is case-sensitive, and must not contain the prefix aws.

DecimalNumber structure

Contains a numeric value in decimal format.

Fields
  • UnscaledValueRequired: Blob.

    The unscaled numeric value.

  • ScaleRequired: Number (integer).

    The scale that determines where the decimal point falls in the unscaled value.

ErrorDetail structure

Contains details about an error.

Fields

PropertyPredicate structure

Defines a property predicate.

Fields
  • Key – Value string, not more than 1024 bytes long.

    The key of the property.

  • Value – Value string, not more than 1024 bytes long.

    The value of the property.

  • Comparator – UTF-8 string (valid values: EQUALS | GREATER_THAN | LESS_THAN | GREATER_THAN_EQUALS | LESS_THAN_EQUALS).

    The comparator used to compare this property to others.

ResourceUri structure

The URIs for function resources.

Fields
  • ResourceType – UTF-8 string (valid values: JAR | FILE | ARCHIVE).

    The type of the resource.

  • Uri – Uniform resource identifier (uri), not less than 1 or more than 1024 bytes long, matching the URI address multi-line string pattern.

    The URI for accessing the resource.

ColumnStatistics structure

Represents the generated column-level statistics for a table or partition.

Fields
  • ColumnNameRequired: UTF-8 string, not less than 1 or more than 255 bytes long, matching the Single-line string pattern.

    Name of column which statistics belong to.

  • ColumnTypeRequired: Type name, not more than 20000 bytes long, matching the Single-line string pattern.

    The data type of the column.

  • AnalyzedTimeRequired: Timestamp.

    The timestamp of when column statistics were generated.

  • StatisticsDataRequired: A ColumnStatisticsData object.

    A ColumnStatisticData object that contains the statistics data values.

ColumnStatisticsError structure

Encapsulates a ColumnStatistics object that failed and the reason for failure.

Fields
  • ColumnStatistics – A ColumnStatistics object.

    The ColumnStatistics of the column.

  • Error – An ErrorDetail object.

    An error message with the reason for the failure of an operation.

ColumnError structure

Encapsulates a column name that failed and the reason for failure.

Fields
  • ColumnName – UTF-8 string, not less than 1 or more than 255 bytes long, matching the Single-line string pattern.

    The name of the column that failed.

  • Error – An ErrorDetail object.

    An error message with the reason for the failure of an operation.

ColumnStatisticsData structure

Contains the individual types of column statistics data. Only one data object should be set and indicated by the Type attribute.

Fields
  • TypeRequired: UTF-8 string (valid values: BOOLEAN | DATE | DECIMAL | DOUBLE | LONG | STRING | BINARY).

    The type of column statistics data.

  • BooleanColumnStatisticsData – A BooleanColumnStatisticsData object.

    Boolean column statistics data.

  • DateColumnStatisticsData – A DateColumnStatisticsData object.

    Date column statistics data.

  • DecimalColumnStatisticsData – A DecimalColumnStatisticsData object.

    Decimal column statistics data. UnscaledValues within are Base64-encoded binary objects storing big-endian, two's complement representations of the decimal's unscaled value.

  • DoubleColumnStatisticsData – A DoubleColumnStatisticsData object.

    Double column statistics data.

  • LongColumnStatisticsData – A LongColumnStatisticsData object.

    Long column statistics data.

  • StringColumnStatisticsData – A StringColumnStatisticsData object.

    String column statistics data.

  • BinaryColumnStatisticsData – A BinaryColumnStatisticsData object.

    Binary column statistics data.

BooleanColumnStatisticsData structure

Defines column statistics supported for Boolean data columns.

Fields
  • NumberOfTruesRequired: Number (long), not more than None.

    The number of true values in the column.

  • NumberOfFalsesRequired: Number (long), not more than None.

    The number of false values in the column.

  • NumberOfNullsRequired: Number (long), not more than None.

    The number of null values in the column.

DateColumnStatisticsData structure

Defines column statistics supported for timestamp data columns.

Fields
  • MinimumValue – Timestamp.

    The lowest value in the column.

  • MaximumValue – Timestamp.

    The highest value in the column.

  • NumberOfNullsRequired: Number (long), not more than None.

    The number of null values in the column.

  • NumberOfDistinctValuesRequired: Number (long), not more than None.

    The number of distinct values in a column.

DecimalColumnStatisticsData structure

Defines column statistics supported for fixed-point number data columns.

Fields
  • MinimumValue – A DecimalNumber object.

    The lowest value in the column.

  • MaximumValue – A DecimalNumber object.

    The highest value in the column.

  • NumberOfNullsRequired: Number (long), not more than None.

    The number of null values in the column.

  • NumberOfDistinctValuesRequired: Number (long), not more than None.

    The number of distinct values in a column.

DoubleColumnStatisticsData structure

Defines column statistics supported for floating-point number data columns.

Fields
  • MinimumValue – Number (double).

    The lowest value in the column.

  • MaximumValue – Number (double).

    The highest value in the column.

  • NumberOfNullsRequired: Number (long), not more than None.

    The number of null values in the column.

  • NumberOfDistinctValuesRequired: Number (long), not more than None.

    The number of distinct values in a column.

LongColumnStatisticsData structure

Defines column statistics supported for integer data columns.

Fields
  • MinimumValue – Number (long).

    The lowest value in the column.

  • MaximumValue – Number (long).

    The highest value in the column.

  • NumberOfNullsRequired: Number (long), not more than None.

    The number of null values in the column.

  • NumberOfDistinctValuesRequired: Number (long), not more than None.

    The number of distinct values in a column.

StringColumnStatisticsData structure

Defines column statistics supported for character sequence data values.

Fields
  • MaximumLengthRequired: Number (long), not more than None.

    The size of the longest string in the column.

  • AverageLengthRequired: Number (double), not more than None.

    The average string length in the column.

  • NumberOfNullsRequired: Number (long), not more than None.

    The number of null values in the column.

  • NumberOfDistinctValuesRequired: Number (long), not more than None.

    The number of distinct values in a column.

BinaryColumnStatisticsData structure

Defines column statistics supported for bit sequence data values.

Fields
  • MaximumLengthRequired: Number (long), not more than None.

    The size of the longest bit sequence in the column.

  • AverageLengthRequired: Number (double), not more than None.

    The average bit sequence length in the column.

  • NumberOfNullsRequired: Number (long), not more than None.

    The number of null values in the column.

String patterns

The API uses the following regular expressions to define what is valid content for various string parameters and members:

  • Single-line string pattern – "[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\t]*"

  • URI address multi-line string pattern – "[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*"

  • A Logstash Grok string pattern – "[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\t]*"

  • Identifier string pattern – "[A-Za-z_][A-Za-z0-9_]*"

  • AWS IAM ARN string pattern – "arn:aws:iam::\d{12}:role/.*"

  • Version string pattern – "^[a-zA-Z0-9-_]+$"

  • Log group string pattern – "[\.\-_/#A-Za-z0-9]+"

  • Log-stream string pattern – "[^:*]*"

  • Custom string pattern #10 – "[^\r\n]"

  • Custom string pattern #11 – "[\p{L}\p{N}\p{P}]*"

  • Custom string pattern #12 – "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}"

  • Custom string pattern #13 – "[a-zA-Z0-9-_$#.]+"

  • Custom string pattern #14 – "^\w+\.\w+\.\w+$"

  • Custom string pattern #15 – "^\w+\.\w+$"

  • Custom string pattern #16 – "^([2-3]|3[.]9)$"

  • Custom string pattern #17 – "arn:(aws|aws-us-gov|aws-cn):glue:.*"

  • Custom string pattern #18 – "(^arn:aws:iam::\w{12}:root)"

  • Custom string pattern #19 – "^arn:aws(-(cn|us-gov|iso(-[bef])?))?:iam::[0-9]{12}:role/.+"

  • Custom string pattern #20 – "arn:aws:kms:.*"

  • Custom string pattern #21 – "arn:aws[^:]*:iam::[0-9]*:role/.+"

  • Custom string pattern #22 – "[\.\-_A-Za-z0-9]+"

  • Custom string pattern #23 – "^s3://([^/]+)/([^/]+/)*([^/]+)$"

  • Custom string pattern #24 – ".*"

  • Custom string pattern #25 – "[a-zA-Z0-9_.-]+"

  • Custom string pattern #26 – ".*\S.*"

  • Custom string pattern #27 – "[a-zA-Z0-9-=._/@]+"

  • Custom string pattern #28 – "[1-9][0-9]*|[1-9][0-9]*-[1-9][0-9]*"

  • Custom string pattern #29 – "[\s\S]*"

  • Custom string pattern #30 – "([\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF]|[^\S\r\n"'= ;])*"

  • Custom string pattern #31 – "[*A-Za-z0-9_-]*"

  • Custom string pattern #32 – "([\u0020-\u007E\r\s\n])*"

  • Custom string pattern #33 – "[A-Za-z0-9_-]*"

  • Custom string pattern #34 – "([\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF]|[^\S\r\n"'])*"

  • Custom string pattern #35 – "([\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF]|[^\S\r\n])*"

  • Custom string pattern #36 – "([\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\s])*"

  • Custom string pattern #37 – "([\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF]|[^\r\n])*"