FlatMap 类

对集合中的每个 DynamicFrame 应用转换。结果不会拼凑成单个 DynamicFrame，而是作为一个集合保存。

FlatMap 的示例

以下示例片段展示了当应用于 FlatMap 时，如何对动态帧集合使用 ResolveChoice 转换。用于输入的数据是位于占位符 Amazon S3 地址 s3://bucket/path-for-data/sample.json 的 JSON，包含以下数据。

JSON 数据示例

[{
    "firstname": "Arnav",
    "lastname": "Desai",
    "address": {
        "street": "6 Anyroad Avenue",
        "city": "London",
        "state": "England",
        "country": "UK"
    },
    "phone": 17235550101,
    "affiliations": [
        "General Anonymous Example Products",
        "Example Independent Research",
        "Government Department of Examples"
    ]
},
{
    "firstname": "Mary",
    "lastname": "Major",
    "address": {
        "street": "7821 Spot Place",
        "city": "Centerville",
        "state": "OK",
        "country": "US"
    },
    "phone": 19185550023,
    "affiliations": [
        "Example Dot Com",
        "Example Independent Research",
        "Example.io"
    ]
},
{
    "firstname": "Paulo",
    "lastname": "Santos",
    "address": {
        "street": "123 Maple Street",
        "city": "London",
        "state": "Ontario",
        "country": "CA"
    },
    "phone": 12175550181,
    "affiliations": [
        "General Anonymous Example Products",
        "Example Dot Com"
    ]
}]

例 将 ResolveChoice 应用于 DynamicFrameCollection，将显示输出。

#Read DynamicFrame
datasource = glueContext.create_dynamic_frame_from_options("s3", connection_options = {"paths":["s3://bucket/path/to/file/mysamplejson.json"]}, format="json")
datasource.printSchema()
datasource.show()

## Split to create a DynamicFrameCollection
split_frame=datasource.split_fields(["firstname","lastname","address"],"personal_info","business_info")
split_frame.keys()
print("---")

## Use FlatMap to run ResolveChoice
kwargs = {"choice": "cast:string"}
flat = FlatMap.apply(split_frame, ResolveChoice, frame_name="frame", transformation_ctx='tcx', **kwargs)
flat.keys()

##Select one of the DynamicFrames
personal_info = flat.select("personal_info")
personal_info.printSchema()
personal_info.show()
print("---")

business_info = flat.select("business_info")
business_info.printSchema()
business_info.show()

重要

调用 FlatMap.apply 时，frame_name 参数必须为 "frame"。目前不接受其他值。

示例输出

root
|-- firstname: string
|-- lastname: string
|-- address: struct
|    |-- street: string
|    |-- city: string
|    |-- state: string
|    |-- country: string
|-- phone: long
|-- affiliations: array
|    |-- element: string
---
{
    "firstname": "Mary",
    "lastname": "Major",
    "address": {
        "street": "7821 Spot Place",
        "city": "Centerville",
        "state": "OK",
        "country": "US"
    },
    "phone": 19185550023,
    "affiliations": [
        "Example Dot Com",
        "Example Independent Research",
        "Example.io"
    ]
}

{
    "firstname": "Paulo",
    "lastname": "Santos",
    "address": {
        "street": "123 Maple Street",
        "city": "London",
        "state": "Ontario",
        "country": "CA"
    },
    "phone": 12175550181,
    "affiliations": [
        "General Anonymous Example Products",
        "Example Dot Com"
    ]
}
---
root
|-- firstname: string
|-- lastname: string
|-- address: struct
|    |-- street: string
|    |-- city: string
|    |-- state: string
|    |-- country: string

{
    "firstname": "Mary",
    "lastname": "Major",
    "address": {
        "street": "7821 Spot Place",
        "city": "Centerville",
        "state": "OK",
        "country": "US"
    }
}

{
    "firstname": "Paulo",
    "lastname": "Santos",
    "address": {
        "street": "123 Maple Street",
        "city": "London",
        "state": "Ontario",
        "country": "CA"
    }
}
---
root
|-- phone: long
|-- affiliations: array
|    |-- element: string

{
    "phone": 19185550023,
    "affiliations": [
        "Example Dot Com",
        "Example Independent Research",
        "Example.io"
    ]
}

{
    "phone": 12175550181,
    "affiliations": [
        "General Anonymous Example Products",
        "Example Dot Com"
    ]
}

方法

• __call__

• Apply

• 名称

• describeArgs

• describeReturn

• describeTransform

• describeErrors

• 描述

frame_name __call__ (dfc，BaseTransform transformation_ctx = “”，**，base_kwargs)，

向集合中的每个 DynamicFrame 应用转换，并展平结果。

• dfc – 要展平的 DynamicFrameCollection (必需)。

• BaseTransform – 派生自 GlueTransform、要应用于集合的每个成员的转换 (必需)。

• frame_name – 要将集合元素传递到的参数名称 (必需)。

• transformation_ctx – 用于标识状态信息的唯一字符串 (可选)。

• base_kwargs – 要传递到基本转换的参数 (必需)。

通过将返回新 DynamicFrameCollection 创建的每个转换 DynamicFrame 在源 DynamicFrameCollection。

apply(cls, *args, **kwargs)

继承自 GlueTransform apply。

name(cls)

继承自 GlueTransform name。

describeArgs(cls)

继承自 GlueTransform describeArgs。

describeReturn(cls)

继承自 GlueTransform describeReturn。

describeTransform(cls)

继承自 GlueTransform describeTransform。

describeErrors(cls)

继承自 GlueTransform describeErrors。

describe(cls)

继承自 GlueTransform describe。