AWS CLI version 2, the latest major version of AWS CLI, is now stable and recommended for general use. To view this page for the AWS CLI version 2, click here. For more information see the AWS CLI version 2 installation instructions and migration guide.
get-plan
--mapping <value>
--source <value>
[--sinks <value>]
[--location <value>]
[--language <value>]
[--additional-plan-options-map <value>]
[--cli-input-json <value>]
[--generate-cli-skeleton <value>]
[--debug]
[--endpoint-url <value>]
[--no-verify-ssl]
[--no-paginate]
[--output <value>]
[--query <value>]
[--profile <value>]
[--region <value>]
[--version <value>]
[--color <value>]
[--no-sign-request]
[--ca-bundle <value>]
[--cli-read-timeout <value>]
[--cli-connect-timeout <value>]
--mapping
(list)
The list of mappings from a source table to target tables.
(structure)
Defines a mapping.
SourceTable -> (string)
The name of the source table.SourcePath -> (string)
The source path.SourceType -> (string)
The source type.TargetTable -> (string)
The target table.TargetPath -> (string)
The target path.TargetType -> (string)
The target type.
Shorthand Syntax:
SourceTable=string,SourcePath=string,SourceType=string,TargetTable=string,TargetPath=string,TargetType=string ...
JSON Syntax:
[
{
"SourceTable": "string",
"SourcePath": "string",
"SourceType": "string",
"TargetTable": "string",
"TargetPath": "string",
"TargetType": "string"
}
...
]
--source
(structure)
The source table.
DatabaseName -> (string)
The database in which the table metadata resides.TableName -> (string)
The name of the table in question.
Shorthand Syntax:
DatabaseName=string,TableName=string
JSON Syntax:
{
"DatabaseName": "string",
"TableName": "string"
}
--sinks
(list)
The target tables.
(structure)
Specifies a table definition in the Glue Data Catalog.
DatabaseName -> (string)
The database in which the table metadata resides.TableName -> (string)
The name of the table in question.
Shorthand Syntax:
DatabaseName=string,TableName=string ...
JSON Syntax:
[
{
"DatabaseName": "string",
"TableName": "string"
}
...
]
--location
(structure)
The parameters for the mapping.
Jdbc -> (list)
A JDBC location.
(structure)
An argument or property of a node.
Name -> (string)
The name of the argument or property.Value -> (string)
The value of the argument or property.Param -> (boolean)
True if the value is used as a parameter.S3 -> (list)
An Amazon Simple Storage Service (Amazon S3) location.
(structure)
An argument or property of a node.
Name -> (string)
The name of the argument or property.Value -> (string)
The value of the argument or property.Param -> (boolean)
True if the value is used as a parameter.DynamoDB -> (list)
An Amazon DynamoDB table location.
(structure)
An argument or property of a node.
Name -> (string)
The name of the argument or property.Value -> (string)
The value of the argument or property.Param -> (boolean)
True if the value is used as a parameter.
Shorthand Syntax:
Jdbc=[{Name=string,Value=string,Param=boolean},{Name=string,Value=string,Param=boolean}],S3=[{Name=string,Value=string,Param=boolean},{Name=string,Value=string,Param=boolean}],DynamoDB=[{Name=string,Value=string,Param=boolean},{Name=string,Value=string,Param=boolean}]
JSON Syntax:
{
"Jdbc": [
{
"Name": "string",
"Value": "string",
"Param": true|false
}
...
],
"S3": [
{
"Name": "string",
"Value": "string",
"Param": true|false
}
...
],
"DynamoDB": [
{
"Name": "string",
"Value": "string",
"Param": true|false
}
...
]
}
--language
(string)
The programming language of the code to perform the mapping.
Possible values:
PYTHON
SCALA
--additional-plan-options-map
(map)
A map to hold additional optional key-value parameters.
Currently, these key-value pairs are supported:
inferSchema
— Specifies whether to setinferSchema
to true or false for the default script generated by an Glue job. For example, to setinferSchema
to true, pass the following key value pair:--additional-plan-options-map '{"inferSchema":"true"}'
key -> (string)
value -> (string)
Shorthand Syntax:
KeyName1=string,KeyName2=string
JSON Syntax:
{"string": "string"
...}
--cli-input-json
(string)
Performs service operation based on the JSON string provided. The JSON string follows the format provided by --generate-cli-skeleton
. If other arguments are provided on the command line, the CLI values will override the JSON-provided values. It is not possible to pass arbitrary binary values using a JSON-provided value as the string will be taken literally.
--generate-cli-skeleton
(string)
Prints a JSON skeleton to standard output without sending an API request. If provided with no value or the value input
, prints a sample input JSON that can be used as an argument for --cli-input-json
. If provided with the value output
, it validates the command inputs and returns a sample output JSON for that command.
--debug
(boolean)
Turn on debug logging.
--endpoint-url
(string)
Override command's default URL with the given URL.
--no-verify-ssl
(boolean)
By default, the AWS CLI uses SSL when communicating with AWS services. For each SSL connection, the AWS CLI will verify SSL certificates. This option overrides the default behavior of verifying SSL certificates.
--no-paginate
(boolean)
Disable automatic pagination. If automatic pagination is disabled, the AWS CLI will only make one call, for the first page of results.
--output
(string)
The formatting style for command output.
--query
(string)
A JMESPath query to use in filtering the response data.
--profile
(string)
Use a specific profile from your credential file.
--region
(string)
The region to use. Overrides config/env settings.
--version
(string)
Display the version of this tool.
--color
(string)
Turn on/off color output.
--no-sign-request
(boolean)
Do not sign requests. Credentials will not be loaded if this argument is provided.
--ca-bundle
(string)
The CA certificate bundle to use when verifying SSL certificates. Overrides config/env settings.
--cli-read-timeout
(int)
The maximum socket read time in seconds. If the value is set to 0, the socket read will be blocking and not timeout. The default value is 60 seconds.
--cli-connect-timeout
(int)
The maximum socket connect time in seconds. If the value is set to 0, the socket connect will be blocking and not timeout. The default value is 60 seconds.
To use the following examples, you must have the AWS CLI installed and configured. See the Getting started guide in the AWS CLI User Guide for more information.
Unless otherwise stated, all examples have unix-like quotation rules. These examples will need to be adapted to your terminal's quoting rules. See Using quotation marks with strings in the AWS CLI User Guide .
To get the generated code for mapping data from source tables to target tables
The following get-plan
retrieves the generated code for mapping columns from the data source to the data target.
aws glue get-plan --mapping '[ \
{ \
"SourcePath":"sensorid", \
"SourceTable":"anything", \
"SourceType":"int", \
"TargetPath":"sensorid", \
"TargetTable":"anything", \
"TargetType":"int" \
}, \
{ \
"SourcePath":"currenttemperature", \
"SourceTable":"anything", \
"SourceType":"int", \
"TargetPath":"currenttemperature", \
"TargetTable":"anything", \
"TargetType":"int" \
}, \
{ \
"SourcePath":"status", \
"SourceTable":"anything", \
"SourceType":"string", \
"TargetPath":"status", \
"TargetTable":"anything", \
"TargetType":"string" \
}]' \
--source '{ \
"DatabaseName":"tempdb", \
"TableName":"s3-source" \
}' \
--sinks '[ \
{ \
"DatabaseName":"tempdb", \
"TableName":"my-s3-sink" \
}]'
--language "scala"
--endpoint https://glue.us-east-1.amazonaws.com
--output "text"
Output:
import com.amazonaws.services.glue.ChoiceOption
import com.amazonaws.services.glue.GlueContext
import com.amazonaws.services.glue.MappingSpec
import com.amazonaws.services.glue.ResolveSpec
import com.amazonaws.services.glue.errors.CallSite
import com.amazonaws.services.glue.util.GlueArgParser
import com.amazonaws.services.glue.util.Job
import com.amazonaws.services.glue.util.JsonOptions
import org.apache.spark.SparkContext
import scala.collection.JavaConverters._
object GlueApp {
def main(sysArgs: Array[String]) {
val spark: SparkContext = new SparkContext()
val glueContext: GlueContext = new GlueContext(spark)
// @params: [JOB_NAME]
val args = GlueArgParser.getResolvedOptions(sysArgs, Seq("JOB_NAME").toArray)
Job.init(args("JOB_NAME"), glueContext, args.asJava)
// @type: DataSource
// @args: [database = "tempdb", table_name = "s3-source", transformation_ctx = "datasource0"]
// @return: datasource0
// @inputs: []
val datasource0 = glueContext.getCatalogSource(database = "tempdb", tableName = "s3-source", redshiftTmpDir = "", transformationContext = "datasource0").getDynamicFrame()
// @type: ApplyMapping
// @args: [mapping = [("sensorid", "int", "sensorid", "int"), ("currenttemperature", "int", "currenttemperature", "int"), ("status", "string", "status", "string")], transformation_ctx = "applymapping1"]
// @return: applymapping1
// @inputs: [frame = datasource0]
val applymapping1 = datasource0.applyMapping(mappings = Seq(("sensorid", "int", "sensorid", "int"), ("currenttemperature", "int", "currenttemperature", "int"), ("status", "string", "status", "string")), caseSensitive = false, transformationContext = "applymapping1")
// @type: SelectFields
// @args: [paths = ["sensorid", "currenttemperature", "status"], transformation_ctx = "selectfields2"]
// @return: selectfields2
// @inputs: [frame = applymapping1]
val selectfields2 = applymapping1.selectFields(paths = Seq("sensorid", "currenttemperature", "status"), transformationContext = "selectfields2")
// @type: ResolveChoice
// @args: [choice = "MATCH_CATALOG", database = "tempdb", table_name = "my-s3-sink", transformation_ctx = "resolvechoice3"]
// @return: resolvechoice3
// @inputs: [frame = selectfields2]
val resolvechoice3 = selectfields2.resolveChoice(choiceOption = Some(ChoiceOption("MATCH_CATALOG")), database = Some("tempdb"), tableName = Some("my-s3-sink"), transformationContext = "resolvechoice3")
// @type: DataSink
// @args: [database = "tempdb", table_name = "my-s3-sink", transformation_ctx = "datasink4"]
// @return: datasink4
// @inputs: [frame = resolvechoice3]
val datasink4 = glueContext.getCatalogSink(database = "tempdb", tableName = "my-s3-sink", redshiftTmpDir = "", transformationContext = "datasink4").writeDynamicFrame(resolvechoice3)
Job.commit()
}
}
For more information, see Editing Scripts in AWS Glue in the AWS Glue Developer Guide.
PythonScript -> (string)
A Python script to perform the mapping.
ScalaCode -> (string)
The Scala code to perform the mapping.