本文為英文版的機器翻譯版本,如內容有任何歧義或不一致之處,概以英文版為準。
在 Amazon Redshift 中讀取和寫入
下列程式碼範例用 PySpark 於透過資料來源 API 和 SparkSQL,從 Amazon Redshift 資料庫讀取和寫入範例資料。
- Data source API
-
用 PySpark 於使用資料來源 API 從 Amazon Redshift 資料庫讀取和寫入範例資料。
import boto3 from pyspark.sql import SQLContext sc = # existing SparkContext sql_context = SQLContext(sc) url = "jdbc:redshift:iam://redshifthost:5439/database" aws_iam_role_arn = "arn:aws:iam::
accountID
:role/roleName
" df = sql_context.read \ .format("io.github.spark_redshift_community.spark.redshift") \ .option("url",url
) \ .option("dbtable", "tableName
") \ .option("tempdir", "s3://path/for/temp/data
") \ .option("aws_iam_role", "aws_iam_role_arn
") \ .load() df.write \ .format("io.github.spark_redshift_community.spark.redshift") \ .option("url",url
) \ .option("dbtable", "tableName_copy
") \ .option("tempdir", "s3://path/for/temp/data
") \ .option("aws_iam_role", "aws_iam_role_arn
") \ .mode("error") \ .save() - SparkSQL
-
用於 PySpark 使用 SparkSQL 從 Amazon Redshift 資料庫讀取和寫入範例資料。
import boto3 import json import sys import os from pyspark.sql import SparkSession spark = SparkSession \ .builder \ .enableHiveSupport() \ .getOrCreate() url = "jdbc:redshift:iam://redshifthost:5439/database" aws_iam_role_arn = "arn:aws:iam::
accountID
:role/roleName
" bucket = "s3://path/for/temp/data
" tableName = "tableName
" # Redshift table name s = f"""CREATE TABLE IF NOT EXISTS {tableName
} (country string, data string) USING io.github.spark_redshift_community.spark.redshift OPTIONS (dbtable '{tableName
}', tempdir '{bucket
}', url '{url
}', aws_iam_role '{aws_iam_role_arn
}' ); """ spark.sql(s) columns = ["country" ,"data"] data = [("test-country
","test-data
")] df = spark.sparkContext.parallelize(data).toDF(columns) # Insert data into table df.write.insertInto(tableName
, overwrite=False) df = spark.sql(f"SELECT * FROM {tableName
}") df.show()
向 Amazon Redshift 進行身分驗證
考量事項