AWS Glue 2017-03-31
- Client: Aws\Glue\GlueClient
- Service ID: glue
- Version: 2017-03-31
This page describes the parameters and results for the operations of the AWS Glue (2017-03-31), and shows how to use the Aws\Glue\GlueClient object to call the described operations. This documentation is specific to the 2017-03-31 API version of the service.
Operation Summary
Each of the following operations can be created from a client using
$client->getCommand('CommandName'), where "CommandName" is the
name of one of the following operations. Note: a command is a value that
encapsulates an operation and the parameters used to create an HTTP request.
You can also create and send a command immediately using the magic methods
available on a client object: $client->commandName(/* parameters */).
You can send the command asynchronously (returning a promise) by appending the
word "Async" to the operation name: $client->commandNameAsync(/* parameters */).
- BatchCreatePartition ( array $params = [] )
Creates one or more partitions in a batch operation.
- BatchDeleteConnection ( array $params = [] )
Deletes a list of connection definitions from the Data Catalog.
- BatchDeletePartition ( array $params = [] )
Deletes one or more partitions in a batch operation.
- BatchDeleteTable ( array $params = [] )
Deletes multiple tables at once.
- BatchDeleteTableVersion ( array $params = [] )
Deletes a specified batch of versions of a table.
- BatchGetCrawlers ( array $params = [] )
Returns a list of resource metadata for a given list of crawler names.
- BatchGetDevEndpoints ( array $params = [] )
Returns a list of resource metadata for a given list of development endpoint names.
- BatchGetJobs ( array $params = [] )
Returns a list of resource metadata for a given list of job names.
- BatchGetPartition ( array $params = [] )
Retrieves partitions in a batch request.
- BatchGetTriggers ( array $params = [] )
Returns a list of resource metadata for a given list of trigger names.
- BatchGetWorkflows ( array $params = [] )
Returns a list of resource metadata for a given list of workflow names.
- BatchStopJobRun ( array $params = [] )
Stops one or more job runs for a specified job definition.
- CancelMLTaskRun ( array $params = [] )
Cancels (stops) a task run.
- CreateClassifier ( array $params = [] )
Creates a classifier in the user's account.
- CreateConnection ( array $params = [] )
Creates a connection definition in the Data Catalog.
- CreateCrawler ( array $params = [] )
Creates a new crawler with specified targets, role, configuration, and optional schedule.
- CreateDatabase ( array $params = [] )
Creates a new database in a Data Catalog.
- CreateDevEndpoint ( array $params = [] )
Creates a new development endpoint.
- CreateJob ( array $params = [] )
Creates a new job definition.
- CreateMLTransform ( array $params = [] )
Creates an AWS Glue machine learning transform.
- CreatePartition ( array $params = [] )
Creates a new partition.
- CreateScript ( array $params = [] )
Transforms a directed acyclic graph (DAG) into code.
- CreateSecurityConfiguration ( array $params = [] )
Creates a new security configuration.
- CreateTable ( array $params = [] )
Creates a new table definition in the Data Catalog.
- CreateTrigger ( array $params = [] )
Creates a new trigger.
- CreateUserDefinedFunction ( array $params = [] )
Creates a new function definition in the Data Catalog.
- CreateWorkflow ( array $params = [] )
Creates a new workflow.
- DeleteClassifier ( array $params = [] )
Removes a classifier from the Data Catalog.
- DeleteConnection ( array $params = [] )
Deletes a connection from the Data Catalog.
- DeleteCrawler ( array $params = [] )
Removes a specified crawler from the AWS Glue Data Catalog, unless the crawler state is RUNNING.
- DeleteDatabase ( array $params = [] )
Removes a specified database from a Data Catalog.
- DeleteDevEndpoint ( array $params = [] )
Deletes a specified development endpoint.
- DeleteJob ( array $params = [] )
Deletes a specified job definition.
- DeleteMLTransform ( array $params = [] )
Deletes an AWS Glue machine learning transform.
- DeletePartition ( array $params = [] )
Deletes a specified partition.
- DeleteResourcePolicy ( array $params = [] )
Deletes a specified policy.
- DeleteSecurityConfiguration ( array $params = [] )
Deletes a specified security configuration.
- DeleteTable ( array $params = [] )
Removes a table definition from the Data Catalog.
- DeleteTableVersion ( array $params = [] )
Deletes a specified version of a table.
- DeleteTrigger ( array $params = [] )
Deletes a specified trigger.
- DeleteUserDefinedFunction ( array $params = [] )
Deletes an existing function definition from the Data Catalog.
- DeleteWorkflow ( array $params = [] )
Deletes a workflow.
- GetCatalogImportStatus ( array $params = [] )
Retrieves the status of a migration operation.
- GetClassifier ( array $params = [] )
Retrieve a classifier by name.
- GetClassifiers ( array $params = [] )
Lists all classifier objects in the Data Catalog.
- GetConnection ( array $params = [] )
Retrieves a connection definition from the Data Catalog.
- GetConnections ( array $params = [] )
Retrieves a list of connection definitions from the Data Catalog.
- GetCrawler ( array $params = [] )
Retrieves metadata for a specified crawler.
- GetCrawlerMetrics ( array $params = [] )
Retrieves metrics about specified crawlers.
- GetCrawlers ( array $params = [] )
Retrieves metadata for all crawlers defined in the customer account.
- GetDataCatalogEncryptionSettings ( array $params = [] )
Retrieves the security configuration for a specified catalog.
- GetDatabase ( array $params = [] )
Retrieves the definition of a specified database.
- GetDatabases ( array $params = [] )
Retrieves all databases defined in a given Data Catalog.
- GetDataflowGraph ( array $params = [] )
Transforms a Python script into a directed acyclic graph (DAG).
- GetDevEndpoint ( array $params = [] )
Retrieves information about a specified development endpoint.
- GetDevEndpoints ( array $params = [] )
Retrieves all the development endpoints in this AWS account.
- GetJob ( array $params = [] )
Retrieves an existing job definition.
- GetJobBookmark ( array $params = [] )
Returns information on a job bookmark entry.
- GetJobRun ( array $params = [] )
Retrieves the metadata for a given job run.
- GetJobRuns ( array $params = [] )
Retrieves metadata for all runs of a given job definition.
- GetJobs ( array $params = [] )
Retrieves all current job definitions.
- GetMLTaskRun ( array $params = [] )
Gets details for a specific task run on a machine learning transform.
- GetMLTaskRuns ( array $params = [] )
Gets a list of runs for a machine learning transform.
- GetMLTransform ( array $params = [] )
Gets an AWS Glue machine learning transform artifact and all its corresponding metadata.
- GetMLTransforms ( array $params = [] )
Gets a sortable, filterable list of existing AWS Glue machine learning transforms.
- GetMapping ( array $params = [] )
Creates mappings.
- GetPartition ( array $params = [] )
Retrieves information about a specified partition.
- GetPartitions ( array $params = [] )
Retrieves information about the partitions in a table.
- GetPlan ( array $params = [] )
Gets code to perform a specified mapping.
- GetResourcePolicy ( array $params = [] )
Retrieves a specified resource policy.
- GetSecurityConfiguration ( array $params = [] )
Retrieves a specified security configuration.
- GetSecurityConfigurations ( array $params = [] )
Retrieves a list of all security configurations.
- GetTable ( array $params = [] )
Retrieves the Table definition in a Data Catalog for a specified table.
- GetTableVersion ( array $params = [] )
Retrieves a specified version of a table.
- GetTableVersions ( array $params = [] )
Retrieves a list of strings that identify available versions of a specified table.
- GetTables ( array $params = [] )
Retrieves the definitions of some or all of the tables in a given Database.
- GetTags ( array $params = [] )
Retrieves a list of tags associated with a resource.
- GetTrigger ( array $params = [] )
Retrieves the definition of a trigger.
- GetTriggers ( array $params = [] )
Gets all the triggers associated with a job.
- GetUserDefinedFunction ( array $params = [] )
Retrieves a specified function definition from the Data Catalog.
- GetUserDefinedFunctions ( array $params = [] )
Retrieves multiple function definitions from the Data Catalog.
- GetWorkflow ( array $params = [] )
Retrieves resource metadata for a workflow.
- GetWorkflowRun ( array $params = [] )
Retrieves the metadata for a given workflow run.
- GetWorkflowRunProperties ( array $params = [] )
Retrieves the workflow run properties which were set during the run.
- GetWorkflowRuns ( array $params = [] )
Retrieves metadata for all runs of a given workflow.
- ImportCatalogToGlue ( array $params = [] )
Imports an existing Amazon Athena Data Catalog to AWS Glue
- ListCrawlers ( array $params = [] )
Retrieves the names of all crawler resources in this AWS account, or the resources with the specified tag.
- ListDevEndpoints ( array $params = [] )
Retrieves the names of all DevEndpoint resources in this AWS account, or the resources with the specified tag.
- ListJobs ( array $params = [] )
Retrieves the names of all job resources in this AWS account, or the resources with the specified tag.
- ListTriggers ( array $params = [] )
Retrieves the names of all trigger resources in this AWS account, or the resources with the specified tag.
- ListWorkflows ( array $params = [] )
Lists names of workflows created in the account.
- PutDataCatalogEncryptionSettings ( array $params = [] )
Sets the security configuration for a specified catalog.
- PutResourcePolicy ( array $params = [] )
Sets the Data Catalog resource policy for access control.
- PutWorkflowRunProperties ( array $params = [] )
Puts the specified workflow run properties for the given workflow run.
- ResetJobBookmark ( array $params = [] )
Resets a bookmark entry.
- SearchTables ( array $params = [] )
Searches a set of tables based on properties in the table metadata as well as on the parent database.
- StartCrawler ( array $params = [] )
Starts a crawl using the specified crawler, regardless of what is scheduled.
- StartCrawlerSchedule ( array $params = [] )
Changes the schedule state of the specified crawler to SCHEDULED, unless the crawler is already running or the schedule state is already SCHEDULED.
- StartExportLabelsTaskRun ( array $params = [] )
Begins an asynchronous task to export all labeled data for a particular transform.
- StartImportLabelsTaskRun ( array $params = [] )
Enables you to provide additional labels (examples of truth) to be used to teach the machine learning transform and improve its quality.
- StartJobRun ( array $params = [] )
Starts a job run using a job definition.
- StartMLEvaluationTaskRun ( array $params = [] )
Starts a task to estimate the quality of the transform.
- StartMLLabelingSetGenerationTaskRun ( array $params = [] )
Starts the active learning workflow for your machine learning transform to improve the transform's quality by generating label sets and adding labels.
- StartTrigger ( array $params = [] )
Starts an existing trigger.
- StartWorkflowRun ( array $params = [] )
Starts a new run of the specified workflow.
- StopCrawler ( array $params = [] )
If the specified crawler is running, stops the crawl.
- StopCrawlerSchedule ( array $params = [] )
Sets the schedule state of the specified crawler to NOT_SCHEDULED, but does not stop the crawler if it is already running.
- StopTrigger ( array $params = [] )
Stops a specified trigger.
- TagResource ( array $params = [] )
Adds tags to a resource.
- UntagResource ( array $params = [] )
Removes tags from a resource.
- UpdateClassifier ( array $params = [] )
Modifies an existing classifier (a GrokClassifier, an XMLClassifier, a JsonClassifier, or a CsvClassifier, depending on which field is present).
- UpdateConnection ( array $params = [] )
Updates a connection definition in the Data Catalog.
- UpdateCrawler ( array $params = [] )
Updates a crawler.
- UpdateCrawlerSchedule ( array $params = [] )
Updates the schedule of a crawler using a cron expression.
- UpdateDatabase ( array $params = [] )
Updates an existing database definition in a Data Catalog.
- UpdateDevEndpoint ( array $params = [] )
Updates a specified development endpoint.
- UpdateJob ( array $params = [] )
Updates an existing job definition.
- UpdateMLTransform ( array $params = [] )
Updates an existing machine learning transform.
- UpdatePartition ( array $params = [] )
Updates a partition.
- UpdateTable ( array $params = [] )
Updates a metadata table in the Data Catalog.
- UpdateTrigger ( array $params = [] )
Updates a trigger definition.
- UpdateUserDefinedFunction ( array $params = [] )
Updates an existing function definition in the Data Catalog.
- UpdateWorkflow ( array $params = [] )
Updates an existing workflow.
Paginators
Paginators handle automatically iterating over paginated API results. Paginators are associated with specific API operations, and they accept the parameters that the corresponding API operation accepts. You can get a paginator from a client class using getPaginator($paginatorName, $operationParameters). This client supports the following paginators:
- GetClassifiers
- GetConnections
- GetCrawlerMetrics
- GetCrawlers
- GetDatabases
- GetDevEndpoints
- GetJobRuns
- GetJobs
- GetMLTaskRuns
- GetMLTransforms
- GetPartitions
- GetSecurityConfigurations
- GetTableVersions
- GetTables
- GetTriggers
- GetUserDefinedFunctions
- GetWorkflowRuns
- ListCrawlers
- ListDevEndpoints
- ListJobs
- ListTriggers
- ListWorkflows
- SearchTables
Operations
BatchCreatePartition
$result = $client->batchCreatePartition([/* ... */]); $promise = $client->batchCreatePartitionAsync([/* ... */]);
Creates one or more partitions in a batch operation.
Parameter Syntax
$result = $client->batchCreatePartition([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'PartitionInputList' => [ // REQUIRED
[
'LastAccessTime' => <integer || string || DateTime>,
'LastAnalyzedTime' => <integer || string || DateTime>,
'Parameters' => ['<string>', ...],
'StorageDescriptor' => [
'BucketColumns' => ['<string>', ...],
'Columns' => [
[
'Comment' => '<string>',
'Name' => '<string>', // REQUIRED
'Parameters' => ['<string>', ...],
'Type' => '<string>',
],
// ...
],
'Compressed' => true || false,
'InputFormat' => '<string>',
'Location' => '<string>',
'NumberOfBuckets' => <integer>,
'OutputFormat' => '<string>',
'Parameters' => ['<string>', ...],
'SerdeInfo' => [
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'SerializationLibrary' => '<string>',
],
'SkewedInfo' => [
'SkewedColumnNames' => ['<string>', ...],
'SkewedColumnValueLocationMaps' => ['<string>', ...],
'SkewedColumnValues' => ['<string>', ...],
],
'SortColumns' => [
[
'Column' => '<string>', // REQUIRED
'SortOrder' => <integer>, // REQUIRED
],
// ...
],
'StoredAsSubDirectories' => true || false,
],
'Values' => ['<string>', ...],
],
// ...
],
'TableName' => '<string>', // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the catalog in which the partition is to be created. Currently, this should be the AWS account ID.
- DatabaseName
-
- Required: Yes
- Type: string
The name of the metadata database in which the partition is to be created.
- PartitionInputList
-
- Required: Yes
- Type: Array of PartitionInput structures
A list of
PartitionInputstructures that define the partitions to be created. - TableName
-
- Required: Yes
- Type: string
The name of the metadata table in which the partition is to be created.
Result Syntax
[
'Errors' => [
[
'ErrorDetail' => [
'ErrorCode' => '<string>',
'ErrorMessage' => '<string>',
],
'PartitionValues' => ['<string>', ...],
],
// ...
],
]
Result Details
Members
- Errors
-
- Type: Array of PartitionError structures
The errors encountered when trying to create the requested partitions.
Errors
-
The input provided was not valid.
-
A resource to be created or added already exists.
-
ResourceNumberLimitExceededException:
A resource numerical limit was exceeded.
-
An internal service error occurred.
-
A specified entity does not exist
-
The operation timed out.
-
An encryption operation failed.
BatchDeleteConnection
$result = $client->batchDeleteConnection([/* ... */]); $promise = $client->batchDeleteConnectionAsync([/* ... */]);
Deletes a list of connection definitions from the Data Catalog.
Parameter Syntax
$result = $client->batchDeleteConnection([
'CatalogId' => '<string>',
'ConnectionNameList' => ['<string>', ...], // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'Errors' => [
'<NameString>' => [
'ErrorCode' => '<string>',
'ErrorMessage' => '<string>',
],
// ...
],
'Succeeded' => ['<string>', ...],
]
Result Details
Members
- Errors
-
- Type: Associative array of custom strings keys (NameString) to ErrorDetail structures
A map of the names of connections that were not successfully deleted to error details.
- Succeeded
-
- Type: Array of strings
A list of names of the connection definitions that were successfully deleted.
Errors
-
An internal service error occurred.
-
The operation timed out.
BatchDeletePartition
$result = $client->batchDeletePartition([/* ... */]); $promise = $client->batchDeletePartitionAsync([/* ... */]);
Deletes one or more partitions in a batch operation.
Parameter Syntax
$result = $client->batchDeletePartition([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'PartitionsToDelete' => [ // REQUIRED
[
'Values' => ['<string>', ...], // REQUIRED
],
// ...
],
'TableName' => '<string>', // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog where the partition to be deleted resides. If none is provided, the AWS account ID is used by default.
- DatabaseName
-
- Required: Yes
- Type: string
The name of the catalog database in which the table in question resides.
- PartitionsToDelete
-
- Required: Yes
- Type: Array of PartitionValueList structures
A list of
PartitionInputstructures that define the partitions to be deleted. - TableName
-
- Required: Yes
- Type: string
The name of the table that contains the partitions to be deleted.
Result Syntax
[
'Errors' => [
[
'ErrorDetail' => [
'ErrorCode' => '<string>',
'ErrorMessage' => '<string>',
],
'PartitionValues' => ['<string>', ...],
],
// ...
],
]
Result Details
Members
- Errors
-
- Type: Array of PartitionError structures
The errors encountered when trying to delete the requested partitions.
Errors
-
The input provided was not valid.
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
BatchDeleteTable
$result = $client->batchDeleteTable([/* ... */]); $promise = $client->batchDeleteTableAsync([/* ... */]);
Deletes multiple tables at once.
After completing this operation, you no longer have access to the table versions and partitions that belong to the deleted table. AWS Glue deletes these "orphaned" resources asynchronously in a timely manner, at the discretion of the service.
To ensure the immediate deletion of all related resources, before calling BatchDeleteTable, use DeleteTableVersion or BatchDeleteTableVersion, and DeletePartition or BatchDeletePartition, to delete any resources that belong to the table.
Parameter Syntax
$result = $client->batchDeleteTable([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'TablesToDelete' => ['<string>', ...], // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog where the table resides. If none is provided, the AWS account ID is used by default.
- DatabaseName
-
- Required: Yes
- Type: string
The name of the catalog database in which the tables to delete reside. For Hive compatibility, this name is entirely lowercase.
- TablesToDelete
-
- Required: Yes
- Type: Array of strings
A list of the table to delete.
Result Syntax
[
'Errors' => [
[
'ErrorDetail' => [
'ErrorCode' => '<string>',
'ErrorMessage' => '<string>',
],
'TableName' => '<string>',
],
// ...
],
]
Result Details
Members
- Errors
-
- Type: Array of TableError structures
A list of errors encountered in attempting to delete the specified tables.
Errors
-
The input provided was not valid.
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
BatchDeleteTableVersion
$result = $client->batchDeleteTableVersion([/* ... */]); $promise = $client->batchDeleteTableVersionAsync([/* ... */]);
Deletes a specified batch of versions of a table.
Parameter Syntax
$result = $client->batchDeleteTableVersion([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'TableName' => '<string>', // REQUIRED
'VersionIds' => ['<string>', ...], // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog where the tables reside. If none is provided, the AWS account ID is used by default.
- DatabaseName
-
- Required: Yes
- Type: string
The database in the catalog in which the table resides. For Hive compatibility, this name is entirely lowercase.
- TableName
-
- Required: Yes
- Type: string
The name of the table. For Hive compatibility, this name is entirely lowercase.
- VersionIds
-
- Required: Yes
- Type: Array of strings
A list of the IDs of versions to be deleted. A
VersionIdis a string representation of an integer. Each version is incremented by 1.
Result Syntax
[
'Errors' => [
[
'ErrorDetail' => [
'ErrorCode' => '<string>',
'ErrorMessage' => '<string>',
],
'TableName' => '<string>',
'VersionId' => '<string>',
],
// ...
],
]
Result Details
Members
- Errors
-
- Type: Array of TableVersionError structures
A list of errors encountered while trying to delete the specified table versions.
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
BatchGetCrawlers
$result = $client->batchGetCrawlers([/* ... */]); $promise = $client->batchGetCrawlersAsync([/* ... */]);
Returns a list of resource metadata for a given list of crawler names. After calling the ListCrawlers operation, you can call this operation to access the data to which you have been granted permissions. This operation supports all IAM permissions, including permission conditions that uses tags.
Parameter Syntax
$result = $client->batchGetCrawlers([
'CrawlerNames' => ['<string>', ...], // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'Crawlers' => [
[
'Classifiers' => ['<string>', ...],
'Configuration' => '<string>',
'CrawlElapsedTime' => <integer>,
'CrawlerSecurityConfiguration' => '<string>',
'CreationTime' => <DateTime>,
'DatabaseName' => '<string>',
'Description' => '<string>',
'LastCrawl' => [
'ErrorMessage' => '<string>',
'LogGroup' => '<string>',
'LogStream' => '<string>',
'MessagePrefix' => '<string>',
'StartTime' => <DateTime>,
'Status' => 'SUCCEEDED|CANCELLED|FAILED',
],
'LastUpdated' => <DateTime>,
'Name' => '<string>',
'Role' => '<string>',
'Schedule' => [
'ScheduleExpression' => '<string>',
'State' => 'SCHEDULED|NOT_SCHEDULED|TRANSITIONING',
],
'SchemaChangePolicy' => [
'DeleteBehavior' => 'LOG|DELETE_FROM_DATABASE|DEPRECATE_IN_DATABASE',
'UpdateBehavior' => 'LOG|UPDATE_IN_DATABASE',
],
'State' => 'READY|RUNNING|STOPPING',
'TablePrefix' => '<string>',
'Targets' => [
'CatalogTargets' => [
[
'DatabaseName' => '<string>',
'Tables' => ['<string>', ...],
],
// ...
],
'DynamoDBTargets' => [
[
'Path' => '<string>',
],
// ...
],
'JdbcTargets' => [
[
'ConnectionName' => '<string>',
'Exclusions' => ['<string>', ...],
'Path' => '<string>',
],
// ...
],
'S3Targets' => [
[
'Exclusions' => ['<string>', ...],
'Path' => '<string>',
],
// ...
],
],
'Version' => <integer>,
],
// ...
],
'CrawlersNotFound' => ['<string>', ...],
]
Result Details
Members
- Crawlers
-
- Type: Array of Crawler structures
A list of crawler definitions.
- CrawlersNotFound
-
- Type: Array of strings
A list of names of crawlers that were not found.
Errors
-
The input provided was not valid.
-
The operation timed out.
BatchGetDevEndpoints
$result = $client->batchGetDevEndpoints([/* ... */]); $promise = $client->batchGetDevEndpointsAsync([/* ... */]);
Returns a list of resource metadata for a given list of development endpoint names. After calling the ListDevEndpoints operation, you can call this operation to access the data to which you have been granted permissions. This operation supports all IAM permissions, including permission conditions that uses tags.
Parameter Syntax
$result = $client->batchGetDevEndpoints([
'DevEndpointNames' => ['<string>', ...], // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'DevEndpoints' => [
[
'Arguments' => ['<string>', ...],
'AvailabilityZone' => '<string>',
'CreatedTimestamp' => <DateTime>,
'EndpointName' => '<string>',
'ExtraJarsS3Path' => '<string>',
'ExtraPythonLibsS3Path' => '<string>',
'FailureReason' => '<string>',
'LastModifiedTimestamp' => <DateTime>,
'LastUpdateStatus' => '<string>',
'NumberOfNodes' => <integer>,
'NumberOfWorkers' => <integer>,
'PrivateAddress' => '<string>',
'PublicAddress' => '<string>',
'PublicKey' => '<string>',
'PublicKeys' => ['<string>', ...],
'RoleArn' => '<string>',
'SecurityConfiguration' => '<string>',
'SecurityGroupIds' => ['<string>', ...],
'Status' => '<string>',
'SubnetId' => '<string>',
'VpcId' => '<string>',
'WorkerType' => 'Standard|G.1X|G.2X',
'YarnEndpointAddress' => '<string>',
'ZeppelinRemoteSparkInterpreterPort' => <integer>,
],
// ...
],
'DevEndpointsNotFound' => ['<string>', ...],
]
Result Details
Members
- DevEndpoints
-
- Type: Array of DevEndpoint structures
A list of
DevEndpointdefinitions. - DevEndpointsNotFound
-
- Type: Array of strings
A list of
DevEndpointsnot found.
Errors
-
Access to a resource was denied.
-
An internal service error occurred.
-
The operation timed out.
-
The input provided was not valid.
BatchGetJobs
$result = $client->batchGetJobs([/* ... */]); $promise = $client->batchGetJobsAsync([/* ... */]);
Returns a list of resource metadata for a given list of job names. After calling the ListJobs operation, you can call this operation to access the data to which you have been granted permissions. This operation supports all IAM permissions, including permission conditions that uses tags.
Parameter Syntax
$result = $client->batchGetJobs([
'JobNames' => ['<string>', ...], // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'Jobs' => [
[
'AllocatedCapacity' => <integer>,
'Command' => [
'Name' => '<string>',
'PythonVersion' => '<string>',
'ScriptLocation' => '<string>',
],
'Connections' => [
'Connections' => ['<string>', ...],
],
'CreatedOn' => <DateTime>,
'DefaultArguments' => ['<string>', ...],
'Description' => '<string>',
'ExecutionProperty' => [
'MaxConcurrentRuns' => <integer>,
],
'GlueVersion' => '<string>',
'LastModifiedOn' => <DateTime>,
'LogUri' => '<string>',
'MaxCapacity' => <float>,
'MaxRetries' => <integer>,
'Name' => '<string>',
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'NumberOfWorkers' => <integer>,
'Role' => '<string>',
'SecurityConfiguration' => '<string>',
'Timeout' => <integer>,
'WorkerType' => 'Standard|G.1X|G.2X',
],
// ...
],
'JobsNotFound' => ['<string>', ...],
]
Result Details
Members
- Jobs
-
- Type: Array of Job structures
A list of job definitions.
- JobsNotFound
-
- Type: Array of strings
A list of names of jobs not found.
Errors
-
An internal service error occurred.
-
The operation timed out.
-
The input provided was not valid.
BatchGetPartition
$result = $client->batchGetPartition([/* ... */]); $promise = $client->batchGetPartitionAsync([/* ... */]);
Retrieves partitions in a batch request.
Parameter Syntax
$result = $client->batchGetPartition([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'PartitionsToGet' => [ // REQUIRED
[
'Values' => ['<string>', ...], // REQUIRED
],
// ...
],
'TableName' => '<string>', // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog where the partitions in question reside. If none is supplied, the AWS account ID is used by default.
- DatabaseName
-
- Required: Yes
- Type: string
The name of the catalog database where the partitions reside.
- PartitionsToGet
-
- Required: Yes
- Type: Array of PartitionValueList structures
A list of partition values identifying the partitions to retrieve.
- TableName
-
- Required: Yes
- Type: string
The name of the partitions' table.
Result Syntax
[
'Partitions' => [
[
'CreationTime' => <DateTime>,
'DatabaseName' => '<string>',
'LastAccessTime' => <DateTime>,
'LastAnalyzedTime' => <DateTime>,
'Parameters' => ['<string>', ...],
'StorageDescriptor' => [
'BucketColumns' => ['<string>', ...],
'Columns' => [
[
'Comment' => '<string>',
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'Type' => '<string>',
],
// ...
],
'Compressed' => true || false,
'InputFormat' => '<string>',
'Location' => '<string>',
'NumberOfBuckets' => <integer>,
'OutputFormat' => '<string>',
'Parameters' => ['<string>', ...],
'SerdeInfo' => [
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'SerializationLibrary' => '<string>',
],
'SkewedInfo' => [
'SkewedColumnNames' => ['<string>', ...],
'SkewedColumnValueLocationMaps' => ['<string>', ...],
'SkewedColumnValues' => ['<string>', ...],
],
'SortColumns' => [
[
'Column' => '<string>',
'SortOrder' => <integer>,
],
// ...
],
'StoredAsSubDirectories' => true || false,
],
'TableName' => '<string>',
'Values' => ['<string>', ...],
],
// ...
],
'UnprocessedKeys' => [
[
'Values' => ['<string>', ...],
],
// ...
],
]
Result Details
Members
- Partitions
-
- Type: Array of Partition structures
A list of the requested partitions.
- UnprocessedKeys
-
- Type: Array of PartitionValueList structures
A list of the partition values in the request for which partitions were not returned.
Errors
-
The input provided was not valid.
-
A specified entity does not exist
-
The operation timed out.
-
An internal service error occurred.
-
An encryption operation failed.
BatchGetTriggers
$result = $client->batchGetTriggers([/* ... */]); $promise = $client->batchGetTriggersAsync([/* ... */]);
Returns a list of resource metadata for a given list of trigger names. After calling the ListTriggers operation, you can call this operation to access the data to which you have been granted permissions. This operation supports all IAM permissions, including permission conditions that uses tags.
Parameter Syntax
$result = $client->batchGetTriggers([
'TriggerNames' => ['<string>', ...], // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'Triggers' => [
[
'Actions' => [
[
'Arguments' => ['<string>', ...],
'CrawlerName' => '<string>',
'JobName' => '<string>',
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'SecurityConfiguration' => '<string>',
'Timeout' => <integer>,
],
// ...
],
'Description' => '<string>',
'Id' => '<string>',
'Name' => '<string>',
'Predicate' => [
'Conditions' => [
[
'CrawlState' => 'RUNNING|SUCCEEDED|CANCELLED|FAILED',
'CrawlerName' => '<string>',
'JobName' => '<string>',
'LogicalOperator' => 'EQUALS',
'State' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
],
// ...
],
'Logical' => 'AND|ANY',
],
'Schedule' => '<string>',
'State' => 'CREATING|CREATED|ACTIVATING|ACTIVATED|DEACTIVATING|DEACTIVATED|DELETING|UPDATING',
'Type' => 'SCHEDULED|CONDITIONAL|ON_DEMAND',
'WorkflowName' => '<string>',
],
// ...
],
'TriggersNotFound' => ['<string>', ...],
]
Result Details
Members
- Triggers
-
- Type: Array of Trigger structures
A list of trigger definitions.
- TriggersNotFound
-
- Type: Array of strings
A list of names of triggers not found.
Errors
-
An internal service error occurred.
-
The operation timed out.
-
The input provided was not valid.
BatchGetWorkflows
$result = $client->batchGetWorkflows([/* ... */]); $promise = $client->batchGetWorkflowsAsync([/* ... */]);
Returns a list of resource metadata for a given list of workflow names. After calling the ListWorkflows operation, you can call this operation to access the data to which you have been granted permissions. This operation supports all IAM permissions, including permission conditions that uses tags.
Parameter Syntax
$result = $client->batchGetWorkflows([
'IncludeGraph' => true || false,
'Names' => ['<string>', ...], // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'MissingWorkflows' => ['<string>', ...],
'Workflows' => [
[
'CreatedOn' => <DateTime>,
'DefaultRunProperties' => ['<string>', ...],
'Description' => '<string>',
'Graph' => [
'Edges' => [
[
'DestinationId' => '<string>',
'SourceId' => '<string>',
],
// ...
],
'Nodes' => [
[
'CrawlerDetails' => [
'Crawls' => [
[
'CompletedOn' => <DateTime>,
'ErrorMessage' => '<string>',
'LogGroup' => '<string>',
'LogStream' => '<string>',
'StartedOn' => <DateTime>,
'State' => 'RUNNING|SUCCEEDED|CANCELLED|FAILED',
],
// ...
],
],
'JobDetails' => [
'JobRuns' => [
[
'AllocatedCapacity' => <integer>,
'Arguments' => ['<string>', ...],
'Attempt' => <integer>,
'CompletedOn' => <DateTime>,
'ErrorMessage' => '<string>',
'ExecutionTime' => <integer>,
'GlueVersion' => '<string>',
'Id' => '<string>',
'JobName' => '<string>',
'JobRunState' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
'LastModifiedOn' => <DateTime>,
'LogGroupName' => '<string>',
'MaxCapacity' => <float>,
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'NumberOfWorkers' => <integer>,
'PredecessorRuns' => [
[
'JobName' => '<string>',
'RunId' => '<string>',
],
// ...
],
'PreviousRunId' => '<string>',
'SecurityConfiguration' => '<string>',
'StartedOn' => <DateTime>,
'Timeout' => <integer>,
'TriggerName' => '<string>',
'WorkerType' => 'Standard|G.1X|G.2X',
],
// ...
],
],
'Name' => '<string>',
'TriggerDetails' => [
'Trigger' => [
'Actions' => [
[
'Arguments' => ['<string>', ...],
'CrawlerName' => '<string>',
'JobName' => '<string>',
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'SecurityConfiguration' => '<string>',
'Timeout' => <integer>,
],
// ...
],
'Description' => '<string>',
'Id' => '<string>',
'Name' => '<string>',
'Predicate' => [
'Conditions' => [
[
'CrawlState' => 'RUNNING|SUCCEEDED|CANCELLED|FAILED',
'CrawlerName' => '<string>',
'JobName' => '<string>',
'LogicalOperator' => 'EQUALS',
'State' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
],
// ...
],
'Logical' => 'AND|ANY',
],
'Schedule' => '<string>',
'State' => 'CREATING|CREATED|ACTIVATING|ACTIVATED|DEACTIVATING|DEACTIVATED|DELETING|UPDATING',
'Type' => 'SCHEDULED|CONDITIONAL|ON_DEMAND',
'WorkflowName' => '<string>',
],
],
'Type' => 'CRAWLER|JOB|TRIGGER',
'UniqueId' => '<string>',
],
// ...
],
],
'LastModifiedOn' => <DateTime>,
'LastRun' => [
'CompletedOn' => <DateTime>,
'Graph' => [
'Edges' => [
[
'DestinationId' => '<string>',
'SourceId' => '<string>',
],
// ...
],
'Nodes' => [
[
'CrawlerDetails' => [
'Crawls' => [
[
'CompletedOn' => <DateTime>,
'ErrorMessage' => '<string>',
'LogGroup' => '<string>',
'LogStream' => '<string>',
'StartedOn' => <DateTime>,
'State' => 'RUNNING|SUCCEEDED|CANCELLED|FAILED',
],
// ...
],
],
'JobDetails' => [
'JobRuns' => [
[
'AllocatedCapacity' => <integer>,
'Arguments' => ['<string>', ...],
'Attempt' => <integer>,
'CompletedOn' => <DateTime>,
'ErrorMessage' => '<string>',
'ExecutionTime' => <integer>,
'GlueVersion' => '<string>',
'Id' => '<string>',
'JobName' => '<string>',
'JobRunState' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
'LastModifiedOn' => <DateTime>,
'LogGroupName' => '<string>',
'MaxCapacity' => <float>,
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'NumberOfWorkers' => <integer>,
'PredecessorRuns' => [
[
'JobName' => '<string>',
'RunId' => '<string>',
],
// ...
],
'PreviousRunId' => '<string>',
'SecurityConfiguration' => '<string>',
'StartedOn' => <DateTime>,
'Timeout' => <integer>,
'TriggerName' => '<string>',
'WorkerType' => 'Standard|G.1X|G.2X',
],
// ...
],
],
'Name' => '<string>',
'TriggerDetails' => [
'Trigger' => [
'Actions' => [
[
'Arguments' => ['<string>', ...],
'CrawlerName' => '<string>',
'JobName' => '<string>',
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'SecurityConfiguration' => '<string>',
'Timeout' => <integer>,
],
// ...
],
'Description' => '<string>',
'Id' => '<string>',
'Name' => '<string>',
'Predicate' => [
'Conditions' => [
[
'CrawlState' => 'RUNNING|SUCCEEDED|CANCELLED|FAILED',
'CrawlerName' => '<string>',
'JobName' => '<string>',
'LogicalOperator' => 'EQUALS',
'State' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
],
// ...
],
'Logical' => 'AND|ANY',
],
'Schedule' => '<string>',
'State' => 'CREATING|CREATED|ACTIVATING|ACTIVATED|DEACTIVATING|DEACTIVATED|DELETING|UPDATING',
'Type' => 'SCHEDULED|CONDITIONAL|ON_DEMAND',
'WorkflowName' => '<string>',
],
],
'Type' => 'CRAWLER|JOB|TRIGGER',
'UniqueId' => '<string>',
],
// ...
],
],
'Name' => '<string>',
'StartedOn' => <DateTime>,
'Statistics' => [
'FailedActions' => <integer>,
'RunningActions' => <integer>,
'StoppedActions' => <integer>,
'SucceededActions' => <integer>,
'TimeoutActions' => <integer>,
'TotalActions' => <integer>,
],
'Status' => 'RUNNING|COMPLETED',
'WorkflowRunId' => '<string>',
'WorkflowRunProperties' => ['<string>', ...],
],
'Name' => '<string>',
],
// ...
],
]
Result Details
Members
- MissingWorkflows
-
- Type: Array of strings
A list of names of workflows not found.
- Workflows
-
- Type: Array of Workflow structures
A list of workflow resource metadata.
Errors
-
An internal service error occurred.
-
The operation timed out.
-
The input provided was not valid.
BatchStopJobRun
$result = $client->batchStopJobRun([/* ... */]); $promise = $client->batchStopJobRunAsync([/* ... */]);
Stops one or more job runs for a specified job definition.
Parameter Syntax
$result = $client->batchStopJobRun([
'JobName' => '<string>', // REQUIRED
'JobRunIds' => ['<string>', ...], // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'Errors' => [
[
'ErrorDetail' => [
'ErrorCode' => '<string>',
'ErrorMessage' => '<string>',
],
'JobName' => '<string>',
'JobRunId' => '<string>',
],
// ...
],
'SuccessfulSubmissions' => [
[
'JobName' => '<string>',
'JobRunId' => '<string>',
],
// ...
],
]
Result Details
Members
- Errors
-
- Type: Array of BatchStopJobRunError structures
A list of the errors that were encountered in trying to stop
JobRuns, including theJobRunIdfor which each error was encountered and details about the error. - SuccessfulSubmissions
-
- Type: Array of BatchStopJobRunSuccessfulSubmission structures
A list of the JobRuns that were successfully submitted for stopping.
Errors
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
CancelMLTaskRun
$result = $client->cancelMLTaskRun([/* ... */]); $promise = $client->cancelMLTaskRunAsync([/* ... */]);
Cancels (stops) a task run. Machine learning task runs are asynchronous tasks that AWS Glue runs on your behalf as part of various machine learning workflows. You can cancel a machine learning task run at any time by calling CancelMLTaskRun with a task run's parent transform's TransformID and the task run's TaskRunId.
Parameter Syntax
$result = $client->cancelMLTaskRun([
'TaskRunId' => '<string>', // REQUIRED
'TransformId' => '<string>', // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'Status' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
'TaskRunId' => '<string>',
'TransformId' => '<string>',
]
Result Details
Members
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
The operation timed out.
-
An internal service error occurred.
CreateClassifier
$result = $client->createClassifier([/* ... */]); $promise = $client->createClassifierAsync([/* ... */]);
Creates a classifier in the user's account. This can be a GrokClassifier, an XMLClassifier, a JsonClassifier, or a CsvClassifier, depending on which field of the request is present.
Parameter Syntax
$result = $client->createClassifier([
'CsvClassifier' => [
'AllowSingleColumn' => true || false,
'ContainsHeader' => 'UNKNOWN|PRESENT|ABSENT',
'Delimiter' => '<string>',
'DisableValueTrimming' => true || false,
'Header' => ['<string>', ...],
'Name' => '<string>', // REQUIRED
'QuoteSymbol' => '<string>',
],
'GrokClassifier' => [
'Classification' => '<string>', // REQUIRED
'CustomPatterns' => '<string>',
'GrokPattern' => '<string>', // REQUIRED
'Name' => '<string>', // REQUIRED
],
'JsonClassifier' => [
'JsonPath' => '<string>', // REQUIRED
'Name' => '<string>', // REQUIRED
],
'XMLClassifier' => [
'Classification' => '<string>', // REQUIRED
'Name' => '<string>', // REQUIRED
'RowTag' => '<string>',
],
]);
Parameter Details
Members
- CsvClassifier
-
- Type: CreateCsvClassifierRequest structure
A
CsvClassifierobject specifying the classifier to create. - GrokClassifier
-
- Type: CreateGrokClassifierRequest structure
A
GrokClassifierobject specifying the classifier to create. - JsonClassifier
-
- Type: CreateJsonClassifierRequest structure
A
JsonClassifierobject specifying the classifier to create. - XMLClassifier
-
- Type: CreateXMLClassifierRequest structure
An
XMLClassifierobject specifying the classifier to create.
Result Syntax
[]
Result Details
Errors
-
A resource to be created or added already exists.
-
The input provided was not valid.
-
The operation timed out.
CreateConnection
$result = $client->createConnection([/* ... */]); $promise = $client->createConnectionAsync([/* ... */]);
Creates a connection definition in the Data Catalog.
Parameter Syntax
$result = $client->createConnection([
'CatalogId' => '<string>',
'ConnectionInput' => [ // REQUIRED
'ConnectionProperties' => ['<string>', ...], // REQUIRED
'ConnectionType' => 'JDBC|SFTP', // REQUIRED
'Description' => '<string>',
'MatchCriteria' => ['<string>', ...],
'Name' => '<string>', // REQUIRED
'PhysicalConnectionRequirements' => [
'AvailabilityZone' => '<string>',
'SecurityGroupIdList' => ['<string>', ...],
'SubnetId' => '<string>',
],
],
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog in which to create the connection. If none is provided, the AWS account ID is used by default.
- ConnectionInput
-
- Required: Yes
- Type: ConnectionInput structure
A
ConnectionInputobject defining the connection to create.
Result Syntax
[]
Result Details
Errors
-
A resource to be created or added already exists.
-
The input provided was not valid.
-
The operation timed out.
-
ResourceNumberLimitExceededException:
A resource numerical limit was exceeded.
-
An encryption operation failed.
CreateCrawler
$result = $client->createCrawler([/* ... */]); $promise = $client->createCrawlerAsync([/* ... */]);
Creates a new crawler with specified targets, role, configuration, and optional schedule. At least one crawl target must be specified, in the s3Targets field, the jdbcTargets field, or the DynamoDBTargets field.
Parameter Syntax
$result = $client->createCrawler([
'Classifiers' => ['<string>', ...],
'Configuration' => '<string>',
'CrawlerSecurityConfiguration' => '<string>',
'DatabaseName' => '<string>',
'Description' => '<string>',
'Name' => '<string>', // REQUIRED
'Role' => '<string>', // REQUIRED
'Schedule' => '<string>',
'SchemaChangePolicy' => [
'DeleteBehavior' => 'LOG|DELETE_FROM_DATABASE|DEPRECATE_IN_DATABASE',
'UpdateBehavior' => 'LOG|UPDATE_IN_DATABASE',
],
'TablePrefix' => '<string>',
'Tags' => ['<string>', ...],
'Targets' => [ // REQUIRED
'CatalogTargets' => [
[
'DatabaseName' => '<string>', // REQUIRED
'Tables' => ['<string>', ...], // REQUIRED
],
// ...
],
'DynamoDBTargets' => [
[
'Path' => '<string>',
],
// ...
],
'JdbcTargets' => [
[
'ConnectionName' => '<string>',
'Exclusions' => ['<string>', ...],
'Path' => '<string>',
],
// ...
],
'S3Targets' => [
[
'Exclusions' => ['<string>', ...],
'Path' => '<string>',
],
// ...
],
],
]);
Parameter Details
Members
- Classifiers
-
- Type: Array of strings
A list of custom classifiers that the user has registered. By default, all built-in classifiers are included in a crawl, but these custom classifiers always override the default classifiers for a given classification.
- Configuration
-
- Type: string
The crawler configuration information. This versioned JSON string allows users to specify aspects of a crawler's behavior. For more information, see Configuring a Crawler.
- CrawlerSecurityConfiguration
-
- Type: string
The name of the
SecurityConfigurationstructure to be used by this crawler. - DatabaseName
-
- Type: string
The AWS Glue database where results are written, such as:
arn:aws:daylight:us-east-1::database/sometable/*. - Description
-
- Type: string
A description of the new crawler.
- Name
-
- Required: Yes
- Type: string
Name of the new crawler.
- Role
-
- Required: Yes
- Type: string
The IAM role or Amazon Resource Name (ARN) of an IAM role used by the new crawler to access customer resources.
- Schedule
-
- Type: string
A
cronexpression used to specify the schedule. For more information, see Time-Based Schedules for Jobs and Crawlers. For example, to run something every day at 12:15 UTC, specifycron(15 12 * * ? *). - SchemaChangePolicy
-
- Type: SchemaChangePolicy structure
The policy for the crawler's update and deletion behavior.
- TablePrefix
-
- Type: string
The table prefix used for catalog tables that are created.
- Tags
-
- Type: Associative array of custom strings keys (TagKey) to strings
The tags to use with this crawler request. You can use tags to limit access to the crawler. For more information, see AWS Tags in AWS Glue.
- Targets
-
- Required: Yes
- Type: CrawlerTargets structure
A list of collection of targets to crawl.
Result Syntax
[]
Result Details
Errors
-
The input provided was not valid.
-
A resource to be created or added already exists.
-
The operation timed out.
-
ResourceNumberLimitExceededException:
A resource numerical limit was exceeded.
CreateDatabase
$result = $client->createDatabase([/* ... */]); $promise = $client->createDatabaseAsync([/* ... */]);
Creates a new database in a Data Catalog.
Parameter Syntax
$result = $client->createDatabase([
'CatalogId' => '<string>',
'DatabaseInput' => [ // REQUIRED
'CreateTableDefaultPermissions' => [
[
'Permissions' => ['<string>', ...],
'Principal' => [
'DataLakePrincipalIdentifier' => '<string>',
],
],
// ...
],
'Description' => '<string>',
'LocationUri' => '<string>',
'Name' => '<string>', // REQUIRED
'Parameters' => ['<string>', ...],
],
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog in which to create the database. If none is provided, the AWS account ID is used by default.
- DatabaseInput
-
- Required: Yes
- Type: DatabaseInput structure
The metadata for the database.
Result Syntax
[]
Result Details
Errors
-
The input provided was not valid.
-
A resource to be created or added already exists.
-
ResourceNumberLimitExceededException:
A resource numerical limit was exceeded.
-
An internal service error occurred.
-
The operation timed out.
-
An encryption operation failed.
CreateDevEndpoint
$result = $client->createDevEndpoint([/* ... */]); $promise = $client->createDevEndpointAsync([/* ... */]);
Creates a new development endpoint.
Parameter Syntax
$result = $client->createDevEndpoint([
'Arguments' => ['<string>', ...],
'EndpointName' => '<string>', // REQUIRED
'ExtraJarsS3Path' => '<string>',
'ExtraPythonLibsS3Path' => '<string>',
'NumberOfNodes' => <integer>,
'NumberOfWorkers' => <integer>,
'PublicKey' => '<string>',
'PublicKeys' => ['<string>', ...],
'RoleArn' => '<string>', // REQUIRED
'SecurityConfiguration' => '<string>',
'SecurityGroupIds' => ['<string>', ...],
'SubnetId' => '<string>',
'Tags' => ['<string>', ...],
'WorkerType' => 'Standard|G.1X|G.2X',
]);
Parameter Details
Members
- Arguments
-
- Type: Associative array of custom strings keys (GenericString) to strings
A map of arguments used to configure the
DevEndpoint. - EndpointName
-
- Required: Yes
- Type: string
The name to be assigned to the new
DevEndpoint. - ExtraJarsS3Path
-
- Type: string
The path to one or more Java
.jarfiles in an S3 bucket that should be loaded in yourDevEndpoint. - ExtraPythonLibsS3Path
-
- Type: string
The paths to one or more Python libraries in an Amazon S3 bucket that should be loaded in your
DevEndpoint. Multiple values must be complete paths separated by a comma.You can only use pure Python libraries with a
DevEndpoint. Libraries that rely on C extensions, such as the pandas Python data analysis library, are not yet supported. - NumberOfNodes
-
- Type: int
The number of AWS Glue Data Processing Units (DPUs) to allocate to this
DevEndpoint. - NumberOfWorkers
-
- Type: int
The number of workers of a defined
workerTypethat are allocated to the development endpoint.The maximum number of workers you can define are 299 for
G.1X, and 149 forG.2X. - PublicKey
-
- Type: string
The public key to be used by this
DevEndpointfor authentication. This attribute is provided for backward compatibility because the recommended attribute to use is public keys. - PublicKeys
-
- Type: Array of strings
A list of public keys to be used by the development endpoints for authentication. The use of this attribute is preferred over a single public key because the public keys allow you to have a different private key per client.
If you previously created an endpoint with a public key, you must remove that key to be able to set a list of public keys. Call the
UpdateDevEndpointAPI with the public key content in thedeletePublicKeysattribute, and the list of new keys in theaddPublicKeysattribute. - RoleArn
-
- Required: Yes
- Type: string
The IAM role for the
DevEndpoint. - SecurityConfiguration
-
- Type: string
The name of the
SecurityConfigurationstructure to be used with thisDevEndpoint. - SecurityGroupIds
-
- Type: Array of strings
Security group IDs for the security groups to be used by the new
DevEndpoint. - SubnetId
-
- Type: string
The subnet ID for the new
DevEndpointto use. - Tags
-
- Type: Associative array of custom strings keys (TagKey) to strings
The tags to use with this DevEndpoint. You may use tags to limit access to the DevEndpoint. For more information about tags in AWS Glue, see AWS Tags in AWS Glue in the developer guide.
- WorkerType
-
- Type: string
The type of predefined worker that is allocated to the development endpoint. Accepts a value of Standard, G.1X, or G.2X.
-
For the
Standardworker type, each worker provides 4 vCPU, 16 GB of memory and a 50GB disk, and 2 executors per worker. -
For the
G.1Xworker type, each worker maps to 1 DPU (4 vCPU, 16 GB of memory, 64 GB disk), and provides 1 executor per worker. We recommend this worker type for memory-intensive jobs. -
For the
G.2Xworker type, each worker maps to 2 DPU (8 vCPU, 32 GB of memory, 128 GB disk), and provides 1 executor per worker. We recommend this worker type for memory-intensive jobs.
Known issue: when a development endpoint is created with the
G.2XWorkerTypeconfiguration, the Spark drivers for the development endpoint will run on 4 vCPU, 16 GB of memory, and a 64 GB disk.
Result Syntax
[
'Arguments' => ['<string>', ...],
'AvailabilityZone' => '<string>',
'CreatedTimestamp' => <DateTime>,
'EndpointName' => '<string>',
'ExtraJarsS3Path' => '<string>',
'ExtraPythonLibsS3Path' => '<string>',
'FailureReason' => '<string>',
'NumberOfNodes' => <integer>,
'NumberOfWorkers' => <integer>,
'RoleArn' => '<string>',
'SecurityConfiguration' => '<string>',
'SecurityGroupIds' => ['<string>', ...],
'Status' => '<string>',
'SubnetId' => '<string>',
'VpcId' => '<string>',
'WorkerType' => 'Standard|G.1X|G.2X',
'YarnEndpointAddress' => '<string>',
'ZeppelinRemoteSparkInterpreterPort' => <integer>,
]
Result Details
Members
- Arguments
-
- Type: Associative array of custom strings keys (GenericString) to strings
The map of arguments used to configure this
DevEndpoint. - AvailabilityZone
-
- Type: string
The AWS Availability Zone where this
DevEndpointis located. - CreatedTimestamp
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The point in time at which this
DevEndpointwas created. - EndpointName
-
- Type: string
The name assigned to the new
DevEndpoint. - ExtraJarsS3Path
-
- Type: string
Path to one or more Java
.jarfiles in an S3 bucket that will be loaded in yourDevEndpoint. - ExtraPythonLibsS3Path
-
- Type: string
The paths to one or more Python libraries in an S3 bucket that will be loaded in your
DevEndpoint. - FailureReason
-
- Type: string
The reason for a current failure in this
DevEndpoint. - NumberOfNodes
-
- Type: int
The number of AWS Glue Data Processing Units (DPUs) allocated to this DevEndpoint.
- NumberOfWorkers
-
- Type: int
The number of workers of a defined
workerTypethat are allocated to the development endpoint. - RoleArn
-
- Type: string
The Amazon Resource Name (ARN) of the role assigned to the new
DevEndpoint. - SecurityConfiguration
-
- Type: string
The name of the
SecurityConfigurationstructure being used with thisDevEndpoint. - SecurityGroupIds
-
- Type: Array of strings
The security groups assigned to the new
DevEndpoint. - Status
-
- Type: string
The current status of the new
DevEndpoint. - SubnetId
-
- Type: string
The subnet ID assigned to the new
DevEndpoint. - VpcId
-
- Type: string
The ID of the virtual private cloud (VPC) used by this
DevEndpoint. - WorkerType
-
- Type: string
The type of predefined worker that is allocated to the development endpoint. May be a value of Standard, G.1X, or G.2X.
- YarnEndpointAddress
-
- Type: string
The address of the YARN endpoint used by this
DevEndpoint. - ZeppelinRemoteSparkInterpreterPort
-
- Type: int
The Apache Zeppelin port for the remote Apache Spark interpreter.
Errors
-
Access to a resource was denied.
-
A resource to be created or added already exists.
-
IdempotentParameterMismatchException:
The same unique identifier was associated with two different records.
-
An internal service error occurred.
-
The operation timed out.
-
The input provided was not valid.
-
A value could not be validated.
-
ResourceNumberLimitExceededException:
A resource numerical limit was exceeded.
CreateJob
$result = $client->createJob([/* ... */]); $promise = $client->createJobAsync([/* ... */]);
Creates a new job definition.
Parameter Syntax
$result = $client->createJob([
'AllocatedCapacity' => <integer>,
'Command' => [ // REQUIRED
'Name' => '<string>',
'PythonVersion' => '<string>',
'ScriptLocation' => '<string>',
],
'Connections' => [
'Connections' => ['<string>', ...],
],
'DefaultArguments' => ['<string>', ...],
'Description' => '<string>',
'ExecutionProperty' => [
'MaxConcurrentRuns' => <integer>,
],
'GlueVersion' => '<string>',
'LogUri' => '<string>',
'MaxCapacity' => <float>,
'MaxRetries' => <integer>,
'Name' => '<string>', // REQUIRED
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'NumberOfWorkers' => <integer>,
'Role' => '<string>', // REQUIRED
'SecurityConfiguration' => '<string>',
'Tags' => ['<string>', ...],
'Timeout' => <integer>,
'WorkerType' => 'Standard|G.1X|G.2X',
]);
Parameter Details
Members
- AllocatedCapacity
-
- Type: int
This parameter is deprecated. Use
MaxCapacityinstead.The number of AWS Glue data processing units (DPUs) to allocate to this Job. You can allocate from 2 to 100 DPUs; the default is 10. A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB of memory. For more information, see the AWS Glue pricing page.
- Command
-
- Required: Yes
- Type: JobCommand structure
The
JobCommandthat executes this job. - Connections
-
- Type: ConnectionsList structure
The connections used for this job.
- DefaultArguments
-
- Type: Associative array of custom strings keys (GenericString) to strings
The default arguments for this job.
You can specify arguments here that your own job-execution script consumes, as well as arguments that AWS Glue itself consumes.
For information about how to specify and consume your own Job arguments, see the Calling AWS Glue APIs in Python topic in the developer guide.
For information about the key-value pairs that AWS Glue consumes to set up your job, see the Special Parameters Used by AWS Glue topic in the developer guide.
- Description
-
- Type: string
Description of the job being defined.
- ExecutionProperty
-
- Type: ExecutionProperty structure
An
ExecutionPropertyspecifying the maximum number of concurrent runs allowed for this job. - GlueVersion
-
- Type: string
Glue version determines the versions of Apache Spark and Python that AWS Glue supports. The Python version indicates the version supported for jobs of type Spark.
For more information about the available AWS Glue versions and corresponding Spark and Python versions, see Glue version in the developer guide.
Jobs that are created without specifying a Glue version default to Glue 0.9.
- LogUri
-
- Type: string
This field is reserved for future use.
- MaxCapacity
-
- Type: double
The number of AWS Glue data processing units (DPUs) that can be allocated when this job runs. A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB of memory. For more information, see the AWS Glue pricing page.
Do not set
Max Capacityif usingWorkerTypeandNumberOfWorkers.The value that can be allocated for
MaxCapacitydepends on whether you are running a Python shell job or an Apache Spark ETL job:-
When you specify a Python shell job (
JobCommand.Name="pythonshell"), you can allocate either 0.0625 or 1 DPU. The default is 0.0625 DPU. -
When you specify an Apache Spark ETL job (
JobCommand.Name="glueetl"), you can allocate from 2 to 100 DPUs. The default is 10 DPUs. This job type cannot have a fractional DPU allocation.
- MaxRetries
-
- Type: int
The maximum number of times to retry this job if it fails.
- Name
-
- Required: Yes
- Type: string
The name you assign to this job definition. It must be unique in your account.
- NotificationProperty
-
- Type: NotificationProperty structure
Specifies configuration properties of a job notification.
- NumberOfWorkers
-
- Type: int
The number of workers of a defined
workerTypethat are allocated when a job runs.The maximum number of workers you can define are 299 for
G.1X, and 149 forG.2X. - Role
-
- Required: Yes
- Type: string
The name or Amazon Resource Name (ARN) of the IAM role associated with this job.
- SecurityConfiguration
-
- Type: string
The name of the
SecurityConfigurationstructure to be used with this job. - Tags
-
- Type: Associative array of custom strings keys (TagKey) to strings
The tags to use with this job. You may use tags to limit access to the job. For more information about tags in AWS Glue, see AWS Tags in AWS Glue in the developer guide.
- Timeout
-
- Type: int
The job timeout in minutes. This is the maximum time that a job run can consume resources before it is terminated and enters
TIMEOUTstatus. The default is 2,880 minutes (48 hours). - WorkerType
-
- Type: string
The type of predefined worker that is allocated when a job runs. Accepts a value of Standard, G.1X, or G.2X.
-
For the
Standardworker type, each worker provides 4 vCPU, 16 GB of memory and a 50GB disk, and 2 executors per worker. -
For the
G.1Xworker type, each worker maps to 1 DPU (4 vCPU, 16 GB of memory, 64 GB disk), and provides 1 executor per worker. We recommend this worker type for memory-intensive jobs. -
For the
G.2Xworker type, each worker maps to 2 DPU (8 vCPU, 32 GB of memory, 128 GB disk), and provides 1 executor per worker. We recommend this worker type for memory-intensive jobs.
Result Syntax
[
'Name' => '<string>',
]
Result Details
Errors
-
The input provided was not valid.
-
IdempotentParameterMismatchException:
The same unique identifier was associated with two different records.
-
A resource to be created or added already exists.
-
An internal service error occurred.
-
The operation timed out.
-
ResourceNumberLimitExceededException:
A resource numerical limit was exceeded.
-
ConcurrentModificationException:
Two processes are trying to modify a resource simultaneously.
CreateMLTransform
$result = $client->createMLTransform([/* ... */]); $promise = $client->createMLTransformAsync([/* ... */]);
Creates an AWS Glue machine learning transform. This operation creates the transform and all the necessary parameters to train it.
Call this operation as the first step in the process of using a machine learning transform (such as the FindMatches transform) for deduplicating data. You can provide an optional Description, in addition to the parameters that you want to use for your algorithm.
You must also specify certain parameters for the tasks that AWS Glue runs on your behalf as part of learning from your data and creating a high-quality machine learning transform. These parameters include Role, and optionally, AllocatedCapacity, Timeout, and MaxRetries. For more information, see Jobs.
Parameter Syntax
$result = $client->createMLTransform([
'Description' => '<string>',
'InputRecordTables' => [ // REQUIRED
[
'CatalogId' => '<string>',
'ConnectionName' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'TableName' => '<string>', // REQUIRED
],
// ...
],
'MaxCapacity' => <float>,
'MaxRetries' => <integer>,
'Name' => '<string>', // REQUIRED
'NumberOfWorkers' => <integer>,
'Parameters' => [ // REQUIRED
'FindMatchesParameters' => [
'AccuracyCostTradeoff' => <float>,
'EnforceProvidedLabels' => true || false,
'PrecisionRecallTradeoff' => <float>,
'PrimaryKeyColumnName' => '<string>',
],
'TransformType' => 'FIND_MATCHES', // REQUIRED
],
'Role' => '<string>', // REQUIRED
'Timeout' => <integer>,
'WorkerType' => 'Standard|G.1X|G.2X',
]);
Parameter Details
Members
- Description
-
- Type: string
A description of the machine learning transform that is being defined. The default is an empty string.
- InputRecordTables
-
- Required: Yes
- Type: Array of GlueTable structures
A list of AWS Glue table definitions used by the transform.
- MaxCapacity
-
- Type: double
The number of AWS Glue data processing units (DPUs) that are allocated to task runs for this transform. You can allocate from 2 to 100 DPUs; the default is 10. A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB of memory. For more information, see the AWS Glue pricing page.
When the
WorkerTypefield is set to a value other thanStandard, theMaxCapacityfield is set automatically and becomes read-only. - MaxRetries
-
- Type: int
The maximum number of times to retry a task for this transform after a task run fails.
- Name
-
- Required: Yes
- Type: string
The unique name that you give the transform when you create it.
- NumberOfWorkers
-
- Type: int
The number of workers of a defined
workerTypethat are allocated when this task runs. - Parameters
-
- Required: Yes
- Type: TransformParameters structure
The algorithmic parameters that are specific to the transform type used. Conditionally dependent on the transform type.
- Role
-
- Required: Yes
- Type: string
The name or Amazon Resource Name (ARN) of the IAM role with the required permissions. Ensure that this role has permission to your Amazon Simple Storage Service (Amazon S3) sources, targets, temporary directory, scripts, and any libraries that are used by the task run for this transform.
- Timeout
-
- Type: int
The timeout of the task run for this transform in minutes. This is the maximum time that a task run for this transform can consume resources before it is terminated and enters
TIMEOUTstatus. The default is 2,880 minutes (48 hours). - WorkerType
-
- Type: string
The type of predefined worker that is allocated when this task runs. Accepts a value of Standard, G.1X, or G.2X.
-
For the
Standardworker type, each worker provides 4 vCPU, 16 GB of memory and a 50GB disk, and 2 executors per worker. -
For the
G.1Xworker type, each worker provides 4 vCPU, 16 GB of memory and a 64GB disk, and 1 executor per worker. -
For the
G.2Xworker type, each worker provides 8 vCPU, 32 GB of memory and a 128GB disk, and 1 executor per worker.
Result Syntax
[
'TransformId' => '<string>',
]
Result Details
Errors
-
A resource to be created or added already exists.
-
The input provided was not valid.
-
The operation timed out.
-
An internal service error occurred.
-
Access to a resource was denied.
-
ResourceNumberLimitExceededException:
A resource numerical limit was exceeded.
-
IdempotentParameterMismatchException:
The same unique identifier was associated with two different records.
CreatePartition
$result = $client->createPartition([/* ... */]); $promise = $client->createPartitionAsync([/* ... */]);
Creates a new partition.
Parameter Syntax
$result = $client->createPartition([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'PartitionInput' => [ // REQUIRED
'LastAccessTime' => <integer || string || DateTime>,
'LastAnalyzedTime' => <integer || string || DateTime>,
'Parameters' => ['<string>', ...],
'StorageDescriptor' => [
'BucketColumns' => ['<string>', ...],
'Columns' => [
[
'Comment' => '<string>',
'Name' => '<string>', // REQUIRED
'Parameters' => ['<string>', ...],
'Type' => '<string>',
],
// ...
],
'Compressed' => true || false,
'InputFormat' => '<string>',
'Location' => '<string>',
'NumberOfBuckets' => <integer>,
'OutputFormat' => '<string>',
'Parameters' => ['<string>', ...],
'SerdeInfo' => [
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'SerializationLibrary' => '<string>',
],
'SkewedInfo' => [
'SkewedColumnNames' => ['<string>', ...],
'SkewedColumnValueLocationMaps' => ['<string>', ...],
'SkewedColumnValues' => ['<string>', ...],
],
'SortColumns' => [
[
'Column' => '<string>', // REQUIRED
'SortOrder' => <integer>, // REQUIRED
],
// ...
],
'StoredAsSubDirectories' => true || false,
],
'Values' => ['<string>', ...],
],
'TableName' => '<string>', // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The AWS account ID of the catalog in which the partition is to be created.
- DatabaseName
-
- Required: Yes
- Type: string
The name of the metadata database in which the partition is to be created.
- PartitionInput
-
- Required: Yes
- Type: PartitionInput structure
A
PartitionInputstructure defining the partition to be created. - TableName
-
- Required: Yes
- Type: string
The name of the metadata table in which the partition is to be created.
Result Syntax
[]
Result Details
Errors
-
The input provided was not valid.
-
A resource to be created or added already exists.
-
ResourceNumberLimitExceededException:
A resource numerical limit was exceeded.
-
An internal service error occurred.
-
A specified entity does not exist
-
The operation timed out.
-
An encryption operation failed.
CreateScript
$result = $client->createScript([/* ... */]); $promise = $client->createScriptAsync([/* ... */]);
Transforms a directed acyclic graph (DAG) into code.
Parameter Syntax
$result = $client->createScript([
'DagEdges' => [
[
'Source' => '<string>', // REQUIRED
'Target' => '<string>', // REQUIRED
'TargetParameter' => '<string>',
],
// ...
],
'DagNodes' => [
[
'Args' => [ // REQUIRED
[
'Name' => '<string>', // REQUIRED
'Param' => true || false,
'Value' => '<string>', // REQUIRED
],
// ...
],
'Id' => '<string>', // REQUIRED
'LineNumber' => <integer>,
'NodeType' => '<string>', // REQUIRED
],
// ...
],
'Language' => 'PYTHON|SCALA',
]);
Parameter Details
Members
- DagEdges
-
- Type: Array of CodeGenEdge structures
A list of the edges in the DAG.
- DagNodes
-
- Type: Array of CodeGenNode structures
A list of the nodes in the DAG.
- Language
-
- Type: string
The programming language of the resulting code from the DAG.
Result Syntax
[
'PythonScript' => '<string>',
'ScalaCode' => '<string>',
]
Result Details
Members
Errors
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
CreateSecurityConfiguration
$result = $client->createSecurityConfiguration([/* ... */]); $promise = $client->createSecurityConfigurationAsync([/* ... */]);
Creates a new security configuration. A security configuration is a set of security properties that can be used by AWS Glue. You can use a security configuration to encrypt data at rest. For information about using security configurations in AWS Glue, see Encrypting Data Written by Crawlers, Jobs, and Development Endpoints.
Parameter Syntax
$result = $client->createSecurityConfiguration([
'EncryptionConfiguration' => [ // REQUIRED
'CloudWatchEncryption' => [
'CloudWatchEncryptionMode' => 'DISABLED|SSE-KMS',
'KmsKeyArn' => '<string>',
],
'JobBookmarksEncryption' => [
'JobBookmarksEncryptionMode' => 'DISABLED|CSE-KMS',
'KmsKeyArn' => '<string>',
],
'S3Encryption' => [
[
'KmsKeyArn' => '<string>',
'S3EncryptionMode' => 'DISABLED|SSE-KMS|SSE-S3',
],
// ...
],
],
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Members
- EncryptionConfiguration
-
- Required: Yes
- Type: EncryptionConfiguration structure
The encryption configuration for the new security configuration.
- Name
-
- Required: Yes
- Type: string
The name for the new security configuration.
Result Syntax
[
'CreatedTimestamp' => <DateTime>,
'Name' => '<string>',
]
Result Details
Members
Errors
-
A resource to be created or added already exists.
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
-
ResourceNumberLimitExceededException:
A resource numerical limit was exceeded.
CreateTable
$result = $client->createTable([/* ... */]); $promise = $client->createTableAsync([/* ... */]);
Creates a new table definition in the Data Catalog.
Parameter Syntax
$result = $client->createTable([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'TableInput' => [ // REQUIRED
'Description' => '<string>',
'LastAccessTime' => <integer || string || DateTime>,
'LastAnalyzedTime' => <integer || string || DateTime>,
'Name' => '<string>', // REQUIRED
'Owner' => '<string>',
'Parameters' => ['<string>', ...],
'PartitionKeys' => [
[
'Comment' => '<string>',
'Name' => '<string>', // REQUIRED
'Parameters' => ['<string>', ...],
'Type' => '<string>',
],
// ...
],
'Retention' => <integer>,
'StorageDescriptor' => [
'BucketColumns' => ['<string>', ...],
'Columns' => [
[
'Comment' => '<string>',
'Name' => '<string>', // REQUIRED
'Parameters' => ['<string>', ...],
'Type' => '<string>',
],
// ...
],
'Compressed' => true || false,
'InputFormat' => '<string>',
'Location' => '<string>',
'NumberOfBuckets' => <integer>,
'OutputFormat' => '<string>',
'Parameters' => ['<string>', ...],
'SerdeInfo' => [
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'SerializationLibrary' => '<string>',
],
'SkewedInfo' => [
'SkewedColumnNames' => ['<string>', ...],
'SkewedColumnValueLocationMaps' => ['<string>', ...],
'SkewedColumnValues' => ['<string>', ...],
],
'SortColumns' => [
[
'Column' => '<string>', // REQUIRED
'SortOrder' => <integer>, // REQUIRED
],
// ...
],
'StoredAsSubDirectories' => true || false,
],
'TableType' => '<string>',
'ViewExpandedText' => '<string>',
'ViewOriginalText' => '<string>',
],
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog in which to create the
Table. If none is supplied, the AWS account ID is used by default. - DatabaseName
-
- Required: Yes
- Type: string
The catalog database in which to create the new table. For Hive compatibility, this name is entirely lowercase.
- TableInput
-
- Required: Yes
- Type: TableInput structure
The
TableInputobject that defines the metadata table to create in the catalog.
Result Syntax
[]
Result Details
Errors
-
A resource to be created or added already exists.
-
The input provided was not valid.
-
A specified entity does not exist
-
ResourceNumberLimitExceededException:
A resource numerical limit was exceeded.
-
An internal service error occurred.
-
The operation timed out.
-
An encryption operation failed.
CreateTrigger
$result = $client->createTrigger([/* ... */]); $promise = $client->createTriggerAsync([/* ... */]);
Creates a new trigger.
Parameter Syntax
$result = $client->createTrigger([
'Actions' => [ // REQUIRED
[
'Arguments' => ['<string>', ...],
'CrawlerName' => '<string>',
'JobName' => '<string>',
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'SecurityConfiguration' => '<string>',
'Timeout' => <integer>,
],
// ...
],
'Description' => '<string>',
'Name' => '<string>', // REQUIRED
'Predicate' => [
'Conditions' => [
[
'CrawlState' => 'RUNNING|SUCCEEDED|CANCELLED|FAILED',
'CrawlerName' => '<string>',
'JobName' => '<string>',
'LogicalOperator' => 'EQUALS',
'State' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
],
// ...
],
'Logical' => 'AND|ANY',
],
'Schedule' => '<string>',
'StartOnCreation' => true || false,
'Tags' => ['<string>', ...],
'Type' => 'SCHEDULED|CONDITIONAL|ON_DEMAND', // REQUIRED
'WorkflowName' => '<string>',
]);
Parameter Details
Members
- Actions
-
- Required: Yes
- Type: Array of Action structures
The actions initiated by this trigger when it fires.
- Description
-
- Type: string
A description of the new trigger.
- Name
-
- Required: Yes
- Type: string
The name of the trigger.
- Predicate
-
- Type: Predicate structure
A predicate to specify when the new trigger should fire.
This field is required when the trigger type is
CONDITIONAL. - Schedule
-
- Type: string
A
cronexpression used to specify the schedule (see Time-Based Schedules for Jobs and Crawlers. For example, to run something every day at 12:15 UTC, you would specify:cron(15 12 * * ? *).This field is required when the trigger type is SCHEDULED.
- StartOnCreation
-
- Type: boolean
Set to
trueto startSCHEDULEDandCONDITIONALtriggers when created. True is not supported forON_DEMANDtriggers. - Tags
-
- Type: Associative array of custom strings keys (TagKey) to strings
The tags to use with this trigger. You may use tags to limit access to the trigger. For more information about tags in AWS Glue, see AWS Tags in AWS Glue in the developer guide.
- Type
-
- Required: Yes
- Type: string
The type of the new trigger.
- WorkflowName
-
- Type: string
The name of the workflow associated with the trigger.
Result Syntax
[
'Name' => '<string>',
]
Result Details
Errors
-
A resource to be created or added already exists.
-
A specified entity does not exist
-
The input provided was not valid.
-
IdempotentParameterMismatchException:
The same unique identifier was associated with two different records.
-
An internal service error occurred.
-
The operation timed out.
-
ResourceNumberLimitExceededException:
A resource numerical limit was exceeded.
-
ConcurrentModificationException:
Two processes are trying to modify a resource simultaneously.
CreateUserDefinedFunction
$result = $client->createUserDefinedFunction([/* ... */]); $promise = $client->createUserDefinedFunctionAsync([/* ... */]);
Creates a new function definition in the Data Catalog.
Parameter Syntax
$result = $client->createUserDefinedFunction([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'FunctionInput' => [ // REQUIRED
'ClassName' => '<string>',
'FunctionName' => '<string>',
'OwnerName' => '<string>',
'OwnerType' => 'USER|ROLE|GROUP',
'ResourceUris' => [
[
'ResourceType' => 'JAR|FILE|ARCHIVE',
'Uri' => '<string>',
],
// ...
],
],
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog in which to create the function. If none is provided, the AWS account ID is used by default.
- DatabaseName
-
- Required: Yes
- Type: string
The name of the catalog database in which to create the function.
- FunctionInput
-
- Required: Yes
- Type: UserDefinedFunctionInput structure
A
FunctionInputobject that defines the function to create in the Data Catalog.
Result Syntax
[]
Result Details
Errors
-
A resource to be created or added already exists.
-
The input provided was not valid.
-
An internal service error occurred.
-
A specified entity does not exist
-
The operation timed out.
-
ResourceNumberLimitExceededException:
A resource numerical limit was exceeded.
-
An encryption operation failed.
CreateWorkflow
$result = $client->createWorkflow([/* ... */]); $promise = $client->createWorkflowAsync([/* ... */]);
Creates a new workflow.
Parameter Syntax
$result = $client->createWorkflow([
'DefaultRunProperties' => ['<string>', ...],
'Description' => '<string>',
'Name' => '<string>', // REQUIRED
'Tags' => ['<string>', ...],
]);
Parameter Details
Members
- DefaultRunProperties
-
- Type: Associative array of custom strings keys (IdString) to strings
A collection of properties to be used as part of each execution of the workflow.
- Description
-
- Type: string
A description of the workflow.
- Name
-
- Required: Yes
- Type: string
The name to be assigned to the workflow. It should be unique within your account.
- Tags
-
- Type: Associative array of custom strings keys (TagKey) to strings
The tags to be used with this workflow.
Result Syntax
[
'Name' => '<string>',
]
Result Details
Errors
-
A resource to be created or added already exists.
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
-
ResourceNumberLimitExceededException:
A resource numerical limit was exceeded.
-
ConcurrentModificationException:
Two processes are trying to modify a resource simultaneously.
DeleteClassifier
$result = $client->deleteClassifier([/* ... */]); $promise = $client->deleteClassifierAsync([/* ... */]);
Removes a classifier from the Data Catalog.
Parameter Syntax
$result = $client->deleteClassifier([
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
The operation timed out.
DeleteConnection
$result = $client->deleteConnection([/* ... */]); $promise = $client->deleteConnectionAsync([/* ... */]);
Deletes a connection from the Data Catalog.
Parameter Syntax
$result = $client->deleteConnection([
'CatalogId' => '<string>',
'ConnectionName' => '<string>', // REQUIRED
]);
Parameter Details
Members
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
The operation timed out.
DeleteCrawler
$result = $client->deleteCrawler([/* ... */]); $promise = $client->deleteCrawlerAsync([/* ... */]);
Removes a specified crawler from the AWS Glue Data Catalog, unless the crawler state is RUNNING.
Parameter Syntax
$result = $client->deleteCrawler([
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
The operation cannot be performed because the crawler is already running.
-
SchedulerTransitioningException:
The specified scheduler is transitioning.
-
The operation timed out.
DeleteDatabase
$result = $client->deleteDatabase([/* ... */]); $promise = $client->deleteDatabaseAsync([/* ... */]);
Removes a specified database from a Data Catalog.
After completing this operation, you no longer have access to the tables (and all table versions and partitions that might belong to the tables) and the user-defined functions in the deleted database. AWS Glue deletes these "orphaned" resources asynchronously in a timely manner, at the discretion of the service.
To ensure the immediate deletion of all related resources, before calling DeleteDatabase, use DeleteTableVersion or BatchDeleteTableVersion, DeletePartition or BatchDeletePartition, DeleteUserDefinedFunction, and DeleteTable or BatchDeleteTable, to delete any resources that belong to the database.
Parameter Syntax
$result = $client->deleteDatabase([
'CatalogId' => '<string>',
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Members
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
DeleteDevEndpoint
$result = $client->deleteDevEndpoint([/* ... */]); $promise = $client->deleteDevEndpointAsync([/* ... */]);
Deletes a specified development endpoint.
Parameter Syntax
$result = $client->deleteDevEndpoint([
'EndpointName' => '<string>', // REQUIRED
]);
Parameter Details
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
-
The input provided was not valid.
DeleteJob
$result = $client->deleteJob([/* ... */]); $promise = $client->deleteJobAsync([/* ... */]);
Deletes a specified job definition. If the job definition is not found, no exception is thrown.
Parameter Syntax
$result = $client->deleteJob([
'JobName' => '<string>', // REQUIRED
]);
Parameter Details
Result Syntax
[
'JobName' => '<string>',
]
Result Details
Errors
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
DeleteMLTransform
$result = $client->deleteMLTransform([/* ... */]); $promise = $client->deleteMLTransformAsync([/* ... */]);
Deletes an AWS Glue machine learning transform. Machine learning transforms are a special type of transform that use machine learning to learn the details of the transformation to be performed by learning from examples provided by humans. These transformations are then saved by AWS Glue. If you no longer need a transform, you can delete it by calling DeleteMLTransforms. However, any AWS Glue jobs that still reference the deleted transform will no longer succeed.
Parameter Syntax
$result = $client->deleteMLTransform([
'TransformId' => '<string>', // REQUIRED
]);
Parameter Details
Result Syntax
[
'TransformId' => '<string>',
]
Result Details
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
The operation timed out.
-
An internal service error occurred.
DeletePartition
$result = $client->deletePartition([/* ... */]); $promise = $client->deletePartitionAsync([/* ... */]);
Deletes a specified partition.
Parameter Syntax
$result = $client->deletePartition([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'PartitionValues' => ['<string>', ...], // REQUIRED
'TableName' => '<string>', // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog where the partition to be deleted resides. If none is provided, the AWS account ID is used by default.
- DatabaseName
-
- Required: Yes
- Type: string
The name of the catalog database in which the table in question resides.
- PartitionValues
-
- Required: Yes
- Type: Array of strings
The values that define the partition.
- TableName
-
- Required: Yes
- Type: string
The name of the table that contains the partition to be deleted.
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
DeleteResourcePolicy
$result = $client->deleteResourcePolicy([/* ... */]); $promise = $client->deleteResourcePolicyAsync([/* ... */]);
Deletes a specified policy.
Parameter Syntax
$result = $client->deleteResourcePolicy([
'PolicyHashCondition' => '<string>',
]);
Parameter Details
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
-
The input provided was not valid.
-
ConditionCheckFailureException:
A specified condition was not satisfied.
DeleteSecurityConfiguration
$result = $client->deleteSecurityConfiguration([/* ... */]); $promise = $client->deleteSecurityConfigurationAsync([/* ... */]);
Deletes a specified security configuration.
Parameter Syntax
$result = $client->deleteSecurityConfiguration([
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
DeleteTable
$result = $client->deleteTable([/* ... */]); $promise = $client->deleteTableAsync([/* ... */]);
Removes a table definition from the Data Catalog.
After completing this operation, you no longer have access to the table versions and partitions that belong to the deleted table. AWS Glue deletes these "orphaned" resources asynchronously in a timely manner, at the discretion of the service.
To ensure the immediate deletion of all related resources, before calling DeleteTable, use DeleteTableVersion or BatchDeleteTableVersion, and DeletePartition or BatchDeletePartition, to delete any resources that belong to the table.
Parameter Syntax
$result = $client->deleteTable([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog where the table resides. If none is provided, the AWS account ID is used by default.
- DatabaseName
-
- Required: Yes
- Type: string
The name of the catalog database in which the table resides. For Hive compatibility, this name is entirely lowercase.
- Name
-
- Required: Yes
- Type: string
The name of the table to be deleted. For Hive compatibility, this name is entirely lowercase.
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
DeleteTableVersion
$result = $client->deleteTableVersion([/* ... */]); $promise = $client->deleteTableVersionAsync([/* ... */]);
Deletes a specified version of a table.
Parameter Syntax
$result = $client->deleteTableVersion([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'TableName' => '<string>', // REQUIRED
'VersionId' => '<string>', // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog where the tables reside. If none is provided, the AWS account ID is used by default.
- DatabaseName
-
- Required: Yes
- Type: string
The database in the catalog in which the table resides. For Hive compatibility, this name is entirely lowercase.
- TableName
-
- Required: Yes
- Type: string
The name of the table. For Hive compatibility, this name is entirely lowercase.
- VersionId
-
- Required: Yes
- Type: string
The ID of the table version to be deleted. A
VersionIDis a string representation of an integer. Each version is incremented by 1.
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
DeleteTrigger
$result = $client->deleteTrigger([/* ... */]); $promise = $client->deleteTriggerAsync([/* ... */]);
Deletes a specified trigger. If the trigger is not found, no exception is thrown.
Parameter Syntax
$result = $client->deleteTrigger([
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Result Syntax
[
'Name' => '<string>',
]
Result Details
Errors
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
-
ConcurrentModificationException:
Two processes are trying to modify a resource simultaneously.
DeleteUserDefinedFunction
$result = $client->deleteUserDefinedFunction([/* ... */]); $promise = $client->deleteUserDefinedFunctionAsync([/* ... */]);
Deletes an existing function definition from the Data Catalog.
Parameter Syntax
$result = $client->deleteUserDefinedFunction([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'FunctionName' => '<string>', // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog where the function to be deleted is located. If none is supplied, the AWS account ID is used by default.
- DatabaseName
-
- Required: Yes
- Type: string
The name of the catalog database where the function is located.
- FunctionName
-
- Required: Yes
- Type: string
The name of the function definition to be deleted.
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
DeleteWorkflow
$result = $client->deleteWorkflow([/* ... */]); $promise = $client->deleteWorkflowAsync([/* ... */]);
Deletes a workflow.
Parameter Syntax
$result = $client->deleteWorkflow([
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Result Syntax
[
'Name' => '<string>',
]
Result Details
Errors
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
-
ConcurrentModificationException:
Two processes are trying to modify a resource simultaneously.
GetCatalogImportStatus
$result = $client->getCatalogImportStatus([/* ... */]); $promise = $client->getCatalogImportStatusAsync([/* ... */]);
Retrieves the status of a migration operation.
Parameter Syntax
$result = $client->getCatalogImportStatus([
'CatalogId' => '<string>',
]);
Parameter Details
Members
Result Syntax
[
'ImportStatus' => [
'ImportCompleted' => true || false,
'ImportTime' => <DateTime>,
'ImportedBy' => '<string>',
],
]
Result Details
Members
- ImportStatus
-
- Type: CatalogImportStatus structure
The status of the specified catalog migration.
Errors
-
An internal service error occurred.
-
The operation timed out.
GetClassifier
$result = $client->getClassifier([/* ... */]); $promise = $client->getClassifierAsync([/* ... */]);
Retrieve a classifier by name.
Parameter Syntax
$result = $client->getClassifier([
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Result Syntax
[
'Classifier' => [
'CsvClassifier' => [
'AllowSingleColumn' => true || false,
'ContainsHeader' => 'UNKNOWN|PRESENT|ABSENT',
'CreationTime' => <DateTime>,
'Delimiter' => '<string>',
'DisableValueTrimming' => true || false,
'Header' => ['<string>', ...],
'LastUpdated' => <DateTime>,
'Name' => '<string>',
'QuoteSymbol' => '<string>',
'Version' => <integer>,
],
'GrokClassifier' => [
'Classification' => '<string>',
'CreationTime' => <DateTime>,
'CustomPatterns' => '<string>',
'GrokPattern' => '<string>',
'LastUpdated' => <DateTime>,
'Name' => '<string>',
'Version' => <integer>,
],
'JsonClassifier' => [
'CreationTime' => <DateTime>,
'JsonPath' => '<string>',
'LastUpdated' => <DateTime>,
'Name' => '<string>',
'Version' => <integer>,
],
'XMLClassifier' => [
'Classification' => '<string>',
'CreationTime' => <DateTime>,
'LastUpdated' => <DateTime>,
'Name' => '<string>',
'RowTag' => '<string>',
'Version' => <integer>,
],
],
]
Result Details
Members
- Classifier
-
- Type: Classifier structure
The requested classifier.
Errors
-
A specified entity does not exist
-
The operation timed out.
GetClassifiers
$result = $client->getClassifiers([/* ... */]); $promise = $client->getClassifiersAsync([/* ... */]);
Lists all classifier objects in the Data Catalog.
Parameter Syntax
$result = $client->getClassifiers([
'MaxResults' => <integer>,
'NextToken' => '<string>',
]);
Parameter Details
Members
Result Syntax
[
'Classifiers' => [
[
'CsvClassifier' => [
'AllowSingleColumn' => true || false,
'ContainsHeader' => 'UNKNOWN|PRESENT|ABSENT',
'CreationTime' => <DateTime>,
'Delimiter' => '<string>',
'DisableValueTrimming' => true || false,
'Header' => ['<string>', ...],
'LastUpdated' => <DateTime>,
'Name' => '<string>',
'QuoteSymbol' => '<string>',
'Version' => <integer>,
],
'GrokClassifier' => [
'Classification' => '<string>',
'CreationTime' => <DateTime>,
'CustomPatterns' => '<string>',
'GrokPattern' => '<string>',
'LastUpdated' => <DateTime>,
'Name' => '<string>',
'Version' => <integer>,
],
'JsonClassifier' => [
'CreationTime' => <DateTime>,
'JsonPath' => '<string>',
'LastUpdated' => <DateTime>,
'Name' => '<string>',
'Version' => <integer>,
],
'XMLClassifier' => [
'Classification' => '<string>',
'CreationTime' => <DateTime>,
'LastUpdated' => <DateTime>,
'Name' => '<string>',
'RowTag' => '<string>',
'Version' => <integer>,
],
],
// ...
],
'NextToken' => '<string>',
]
Result Details
Members
- Classifiers
-
- Type: Array of Classifier structures
The requested list of classifier objects.
- NextToken
-
- Type: string
A continuation token.
Errors
-
The operation timed out.
GetConnection
$result = $client->getConnection([/* ... */]); $promise = $client->getConnectionAsync([/* ... */]);
Retrieves a connection definition from the Data Catalog.
Parameter Syntax
$result = $client->getConnection([
'CatalogId' => '<string>',
'HidePassword' => true || false,
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog in which the connection resides. If none is provided, the AWS account ID is used by default.
- HidePassword
-
- Type: boolean
Allows you to retrieve the connection metadata without returning the password. For instance, the AWS Glue console uses this flag to retrieve the connection, and does not display the password. Set this parameter when the caller might not have permission to use the AWS KMS key to decrypt the password, but it does have permission to access the rest of the connection properties.
- Name
-
- Required: Yes
- Type: string
The name of the connection definition to retrieve.
Result Syntax
[
'Connection' => [
'ConnectionProperties' => ['<string>', ...],
'ConnectionType' => 'JDBC|SFTP',
'CreationTime' => <DateTime>,
'Description' => '<string>',
'LastUpdatedBy' => '<string>',
'LastUpdatedTime' => <DateTime>,
'MatchCriteria' => ['<string>', ...],
'Name' => '<string>',
'PhysicalConnectionRequirements' => [
'AvailabilityZone' => '<string>',
'SecurityGroupIdList' => ['<string>', ...],
'SubnetId' => '<string>',
],
],
]
Result Details
Members
- Connection
-
- Type: Connection structure
The requested connection definition.
Errors
-
A specified entity does not exist
-
The operation timed out.
-
The input provided was not valid.
-
An encryption operation failed.
GetConnections
$result = $client->getConnections([/* ... */]); $promise = $client->getConnectionsAsync([/* ... */]);
Retrieves a list of connection definitions from the Data Catalog.
Parameter Syntax
$result = $client->getConnections([
'CatalogId' => '<string>',
'Filter' => [
'ConnectionType' => 'JDBC|SFTP',
'MatchCriteria' => ['<string>', ...],
],
'HidePassword' => true || false,
'MaxResults' => <integer>,
'NextToken' => '<string>',
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog in which the connections reside. If none is provided, the AWS account ID is used by default.
- Filter
-
- Type: GetConnectionsFilter structure
A filter that controls which connections are returned.
- HidePassword
-
- Type: boolean
Allows you to retrieve the connection metadata without returning the password. For instance, the AWS Glue console uses this flag to retrieve the connection, and does not display the password. Set this parameter when the caller might not have permission to use the AWS KMS key to decrypt the password, but it does have permission to access the rest of the connection properties.
- MaxResults
-
- Type: int
The maximum number of connections to return in one response.
- NextToken
-
- Type: string
A continuation token, if this is a continuation call.
Result Syntax
[
'ConnectionList' => [
[
'ConnectionProperties' => ['<string>', ...],
'ConnectionType' => 'JDBC|SFTP',
'CreationTime' => <DateTime>,
'Description' => '<string>',
'LastUpdatedBy' => '<string>',
'LastUpdatedTime' => <DateTime>,
'MatchCriteria' => ['<string>', ...],
'Name' => '<string>',
'PhysicalConnectionRequirements' => [
'AvailabilityZone' => '<string>',
'SecurityGroupIdList' => ['<string>', ...],
'SubnetId' => '<string>',
],
],
// ...
],
'NextToken' => '<string>',
]
Result Details
Members
- ConnectionList
-
- Type: Array of Connection structures
A list of requested connection definitions.
- NextToken
-
- Type: string
A continuation token, if the list of connections returned does not include the last of the filtered connections.
Errors
-
A specified entity does not exist
-
The operation timed out.
-
The input provided was not valid.
-
An encryption operation failed.
GetCrawler
$result = $client->getCrawler([/* ... */]); $promise = $client->getCrawlerAsync([/* ... */]);
Retrieves metadata for a specified crawler.
Parameter Syntax
$result = $client->getCrawler([
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Result Syntax
[
'Crawler' => [
'Classifiers' => ['<string>', ...],
'Configuration' => '<string>',
'CrawlElapsedTime' => <integer>,
'CrawlerSecurityConfiguration' => '<string>',
'CreationTime' => <DateTime>,
'DatabaseName' => '<string>',
'Description' => '<string>',
'LastCrawl' => [
'ErrorMessage' => '<string>',
'LogGroup' => '<string>',
'LogStream' => '<string>',
'MessagePrefix' => '<string>',
'StartTime' => <DateTime>,
'Status' => 'SUCCEEDED|CANCELLED|FAILED',
],
'LastUpdated' => <DateTime>,
'Name' => '<string>',
'Role' => '<string>',
'Schedule' => [
'ScheduleExpression' => '<string>',
'State' => 'SCHEDULED|NOT_SCHEDULED|TRANSITIONING',
],
'SchemaChangePolicy' => [
'DeleteBehavior' => 'LOG|DELETE_FROM_DATABASE|DEPRECATE_IN_DATABASE',
'UpdateBehavior' => 'LOG|UPDATE_IN_DATABASE',
],
'State' => 'READY|RUNNING|STOPPING',
'TablePrefix' => '<string>',
'Targets' => [
'CatalogTargets' => [
[
'DatabaseName' => '<string>',
'Tables' => ['<string>', ...],
],
// ...
],
'DynamoDBTargets' => [
[
'Path' => '<string>',
],
// ...
],
'JdbcTargets' => [
[
'ConnectionName' => '<string>',
'Exclusions' => ['<string>', ...],
'Path' => '<string>',
],
// ...
],
'S3Targets' => [
[
'Exclusions' => ['<string>', ...],
'Path' => '<string>',
],
// ...
],
],
'Version' => <integer>,
],
]
Result Details
Members
- Crawler
-
- Type: Crawler structure
The metadata for the specified crawler.
Errors
-
A specified entity does not exist
-
The operation timed out.
GetCrawlerMetrics
$result = $client->getCrawlerMetrics([/* ... */]); $promise = $client->getCrawlerMetricsAsync([/* ... */]);
Retrieves metrics about specified crawlers.
Parameter Syntax
$result = $client->getCrawlerMetrics([
'CrawlerNameList' => ['<string>', ...],
'MaxResults' => <integer>,
'NextToken' => '<string>',
]);
Parameter Details
Members
Result Syntax
[
'CrawlerMetricsList' => [
[
'CrawlerName' => '<string>',
'LastRuntimeSeconds' => <float>,
'MedianRuntimeSeconds' => <float>,
'StillEstimating' => true || false,
'TablesCreated' => <integer>,
'TablesDeleted' => <integer>,
'TablesUpdated' => <integer>,
'TimeLeftSeconds' => <float>,
],
// ...
],
'NextToken' => '<string>',
]
Result Details
Members
- CrawlerMetricsList
-
- Type: Array of CrawlerMetrics structures
A list of metrics for the specified crawler.
- NextToken
-
- Type: string
A continuation token, if the returned list does not contain the last metric available.
Errors
-
The operation timed out.
GetCrawlers
$result = $client->getCrawlers([/* ... */]); $promise = $client->getCrawlersAsync([/* ... */]);
Retrieves metadata for all crawlers defined in the customer account.
Parameter Syntax
$result = $client->getCrawlers([
'MaxResults' => <integer>,
'NextToken' => '<string>',
]);
Parameter Details
Members
Result Syntax
[
'Crawlers' => [
[
'Classifiers' => ['<string>', ...],
'Configuration' => '<string>',
'CrawlElapsedTime' => <integer>,
'CrawlerSecurityConfiguration' => '<string>',
'CreationTime' => <DateTime>,
'DatabaseName' => '<string>',
'Description' => '<string>',
'LastCrawl' => [
'ErrorMessage' => '<string>',
'LogGroup' => '<string>',
'LogStream' => '<string>',
'MessagePrefix' => '<string>',
'StartTime' => <DateTime>,
'Status' => 'SUCCEEDED|CANCELLED|FAILED',
],
'LastUpdated' => <DateTime>,
'Name' => '<string>',
'Role' => '<string>',
'Schedule' => [
'ScheduleExpression' => '<string>',
'State' => 'SCHEDULED|NOT_SCHEDULED|TRANSITIONING',
],
'SchemaChangePolicy' => [
'DeleteBehavior' => 'LOG|DELETE_FROM_DATABASE|DEPRECATE_IN_DATABASE',
'UpdateBehavior' => 'LOG|UPDATE_IN_DATABASE',
],
'State' => 'READY|RUNNING|STOPPING',
'TablePrefix' => '<string>',
'Targets' => [
'CatalogTargets' => [
[
'DatabaseName' => '<string>',
'Tables' => ['<string>', ...],
],
// ...
],
'DynamoDBTargets' => [
[
'Path' => '<string>',
],
// ...
],
'JdbcTargets' => [
[
'ConnectionName' => '<string>',
'Exclusions' => ['<string>', ...],
'Path' => '<string>',
],
// ...
],
'S3Targets' => [
[
'Exclusions' => ['<string>', ...],
'Path' => '<string>',
],
// ...
],
],
'Version' => <integer>,
],
// ...
],
'NextToken' => '<string>',
]
Result Details
Members
- Crawlers
-
- Type: Array of Crawler structures
A list of crawler metadata.
- NextToken
-
- Type: string
A continuation token, if the returned list has not reached the end of those defined in this customer account.
Errors
-
The operation timed out.
GetDataCatalogEncryptionSettings
$result = $client->getDataCatalogEncryptionSettings([/* ... */]); $promise = $client->getDataCatalogEncryptionSettingsAsync([/* ... */]);
Retrieves the security configuration for a specified catalog.
Parameter Syntax
$result = $client->getDataCatalogEncryptionSettings([
'CatalogId' => '<string>',
]);
Parameter Details
Members
Result Syntax
[
'DataCatalogEncryptionSettings' => [
'ConnectionPasswordEncryption' => [
'AwsKmsKeyId' => '<string>',
'ReturnConnectionPasswordEncrypted' => true || false,
],
'EncryptionAtRest' => [
'CatalogEncryptionMode' => 'DISABLED|SSE-KMS',
'SseAwsKmsKeyId' => '<string>',
],
],
]
Result Details
Members
- DataCatalogEncryptionSettings
-
- Type: DataCatalogEncryptionSettings structure
The requested security configuration.
Errors
-
An internal service error occurred.
-
The input provided was not valid.
-
The operation timed out.
GetDatabase
$result = $client->getDatabase([/* ... */]); $promise = $client->getDatabaseAsync([/* ... */]);
Retrieves the definition of a specified database.
Parameter Syntax
$result = $client->getDatabase([
'CatalogId' => '<string>',
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'Database' => [
'CreateTableDefaultPermissions' => [
[
'Permissions' => ['<string>', ...],
'Principal' => [
'DataLakePrincipalIdentifier' => '<string>',
],
],
// ...
],
'CreateTime' => <DateTime>,
'Description' => '<string>',
'LocationUri' => '<string>',
'Name' => '<string>',
'Parameters' => ['<string>', ...],
],
]
Result Details
Members
- Database
-
- Type: Database structure
The definition of the specified database in the Data Catalog.
Errors
-
The input provided was not valid.
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
-
An encryption operation failed.
GetDatabases
$result = $client->getDatabases([/* ... */]); $promise = $client->getDatabasesAsync([/* ... */]);
Retrieves all databases defined in a given Data Catalog.
Parameter Syntax
$result = $client->getDatabases([
'CatalogId' => '<string>',
'MaxResults' => <integer>,
'NextToken' => '<string>',
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog from which to retrieve
Databases. If none is provided, the AWS account ID is used by default. - MaxResults
-
- Type: int
The maximum number of databases to return in one response.
- NextToken
-
- Type: string
A continuation token, if this is a continuation call.
Result Syntax
[
'DatabaseList' => [
[
'CreateTableDefaultPermissions' => [
[
'Permissions' => ['<string>', ...],
'Principal' => [
'DataLakePrincipalIdentifier' => '<string>',
],
],
// ...
],
'CreateTime' => <DateTime>,
'Description' => '<string>',
'LocationUri' => '<string>',
'Name' => '<string>',
'Parameters' => ['<string>', ...],
],
// ...
],
'NextToken' => '<string>',
]
Result Details
Members
- DatabaseList
-
- Required: Yes
- Type: Array of Database structures
A list of
Databaseobjects from the specified catalog. - NextToken
-
- Type: string
A continuation token for paginating the returned list of tokens, returned if the current segment of the list is not the last.
Errors
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
-
An encryption operation failed.
GetDataflowGraph
$result = $client->getDataflowGraph([/* ... */]); $promise = $client->getDataflowGraphAsync([/* ... */]);
Transforms a Python script into a directed acyclic graph (DAG).
Parameter Syntax
$result = $client->getDataflowGraph([
'PythonScript' => '<string>',
]);
Parameter Details
Result Syntax
[
'DagEdges' => [
[
'Source' => '<string>',
'Target' => '<string>',
'TargetParameter' => '<string>',
],
// ...
],
'DagNodes' => [
[
'Args' => [
[
'Name' => '<string>',
'Param' => true || false,
'Value' => '<string>',
],
// ...
],
'Id' => '<string>',
'LineNumber' => <integer>,
'NodeType' => '<string>',
],
// ...
],
]
Result Details
Members
- DagEdges
-
- Type: Array of CodeGenEdge structures
A list of the edges in the resulting DAG.
- DagNodes
-
- Type: Array of CodeGenNode structures
A list of the nodes in the resulting DAG.
Errors
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
GetDevEndpoint
$result = $client->getDevEndpoint([/* ... */]); $promise = $client->getDevEndpointAsync([/* ... */]);
Retrieves information about a specified development endpoint.
When you create a development endpoint in a virtual private cloud (VPC), AWS Glue returns only a private IP address, and the public IP address field is not populated. When you create a non-VPC development endpoint, AWS Glue returns only a public IP address.
Parameter Syntax
$result = $client->getDevEndpoint([
'EndpointName' => '<string>', // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'DevEndpoint' => [
'Arguments' => ['<string>', ...],
'AvailabilityZone' => '<string>',
'CreatedTimestamp' => <DateTime>,
'EndpointName' => '<string>',
'ExtraJarsS3Path' => '<string>',
'ExtraPythonLibsS3Path' => '<string>',
'FailureReason' => '<string>',
'LastModifiedTimestamp' => <DateTime>,
'LastUpdateStatus' => '<string>',
'NumberOfNodes' => <integer>,
'NumberOfWorkers' => <integer>,
'PrivateAddress' => '<string>',
'PublicAddress' => '<string>',
'PublicKey' => '<string>',
'PublicKeys' => ['<string>', ...],
'RoleArn' => '<string>',
'SecurityConfiguration' => '<string>',
'SecurityGroupIds' => ['<string>', ...],
'Status' => '<string>',
'SubnetId' => '<string>',
'VpcId' => '<string>',
'WorkerType' => 'Standard|G.1X|G.2X',
'YarnEndpointAddress' => '<string>',
'ZeppelinRemoteSparkInterpreterPort' => <integer>,
],
]
Result Details
Members
- DevEndpoint
-
- Type: DevEndpoint structure
A
DevEndpointdefinition.
Errors
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
-
The input provided was not valid.
GetDevEndpoints
$result = $client->getDevEndpoints([/* ... */]); $promise = $client->getDevEndpointsAsync([/* ... */]);
Retrieves all the development endpoints in this AWS account.
When you create a development endpoint in a virtual private cloud (VPC), AWS Glue returns only a private IP address and the public IP address field is not populated. When you create a non-VPC development endpoint, AWS Glue returns only a public IP address.
Parameter Syntax
$result = $client->getDevEndpoints([
'MaxResults' => <integer>,
'NextToken' => '<string>',
]);
Parameter Details
Members
Result Syntax
[
'DevEndpoints' => [
[
'Arguments' => ['<string>', ...],
'AvailabilityZone' => '<string>',
'CreatedTimestamp' => <DateTime>,
'EndpointName' => '<string>',
'ExtraJarsS3Path' => '<string>',
'ExtraPythonLibsS3Path' => '<string>',
'FailureReason' => '<string>',
'LastModifiedTimestamp' => <DateTime>,
'LastUpdateStatus' => '<string>',
'NumberOfNodes' => <integer>,
'NumberOfWorkers' => <integer>,
'PrivateAddress' => '<string>',
'PublicAddress' => '<string>',
'PublicKey' => '<string>',
'PublicKeys' => ['<string>', ...],
'RoleArn' => '<string>',
'SecurityConfiguration' => '<string>',
'SecurityGroupIds' => ['<string>', ...],
'Status' => '<string>',
'SubnetId' => '<string>',
'VpcId' => '<string>',
'WorkerType' => 'Standard|G.1X|G.2X',
'YarnEndpointAddress' => '<string>',
'ZeppelinRemoteSparkInterpreterPort' => <integer>,
],
// ...
],
'NextToken' => '<string>',
]
Result Details
Members
- DevEndpoints
-
- Type: Array of DevEndpoint structures
A list of
DevEndpointdefinitions. - NextToken
-
- Type: string
A continuation token, if not all
DevEndpointdefinitions have yet been returned.
Errors
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
-
The input provided was not valid.
GetJob
$result = $client->getJob([/* ... */]); $promise = $client->getJobAsync([/* ... */]);
Retrieves an existing job definition.
Parameter Syntax
$result = $client->getJob([
'JobName' => '<string>', // REQUIRED
]);
Parameter Details
Result Syntax
[
'Job' => [
'AllocatedCapacity' => <integer>,
'Command' => [
'Name' => '<string>',
'PythonVersion' => '<string>',
'ScriptLocation' => '<string>',
],
'Connections' => [
'Connections' => ['<string>', ...],
],
'CreatedOn' => <DateTime>,
'DefaultArguments' => ['<string>', ...],
'Description' => '<string>',
'ExecutionProperty' => [
'MaxConcurrentRuns' => <integer>,
],
'GlueVersion' => '<string>',
'LastModifiedOn' => <DateTime>,
'LogUri' => '<string>',
'MaxCapacity' => <float>,
'MaxRetries' => <integer>,
'Name' => '<string>',
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'NumberOfWorkers' => <integer>,
'Role' => '<string>',
'SecurityConfiguration' => '<string>',
'Timeout' => <integer>,
'WorkerType' => 'Standard|G.1X|G.2X',
],
]
Result Details
Members
- Job
-
- Type: Job structure
The requested job definition.
Errors
-
The input provided was not valid.
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
GetJobBookmark
$result = $client->getJobBookmark([/* ... */]); $promise = $client->getJobBookmarkAsync([/* ... */]);
Returns information on a job bookmark entry.
Parameter Syntax
$result = $client->getJobBookmark([
'JobName' => '<string>', // REQUIRED
'RunId' => '<string>',
]);
Parameter Details
Members
Result Syntax
[
'JobBookmarkEntry' => [
'Attempt' => <integer>,
'JobBookmark' => '<string>',
'JobName' => '<string>',
'PreviousRunId' => '<string>',
'Run' => <integer>,
'RunId' => '<string>',
'Version' => <integer>,
],
]
Result Details
Members
- JobBookmarkEntry
-
- Type: JobBookmarkEntry structure
A structure that defines a point that a job can resume processing.
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
-
A value could not be validated.
GetJobRun
$result = $client->getJobRun([/* ... */]); $promise = $client->getJobRunAsync([/* ... */]);
Retrieves the metadata for a given job run.
Parameter Syntax
$result = $client->getJobRun([
'JobName' => '<string>', // REQUIRED
'PredecessorsIncluded' => true || false,
'RunId' => '<string>', // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'JobRun' => [
'AllocatedCapacity' => <integer>,
'Arguments' => ['<string>', ...],
'Attempt' => <integer>,
'CompletedOn' => <DateTime>,
'ErrorMessage' => '<string>',
'ExecutionTime' => <integer>,
'GlueVersion' => '<string>',
'Id' => '<string>',
'JobName' => '<string>',
'JobRunState' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
'LastModifiedOn' => <DateTime>,
'LogGroupName' => '<string>',
'MaxCapacity' => <float>,
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'NumberOfWorkers' => <integer>,
'PredecessorRuns' => [
[
'JobName' => '<string>',
'RunId' => '<string>',
],
// ...
],
'PreviousRunId' => '<string>',
'SecurityConfiguration' => '<string>',
'StartedOn' => <DateTime>,
'Timeout' => <integer>,
'TriggerName' => '<string>',
'WorkerType' => 'Standard|G.1X|G.2X',
],
]
Result Details
Members
- JobRun
-
- Type: JobRun structure
The requested job-run metadata.
Errors
-
The input provided was not valid.
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
GetJobRuns
$result = $client->getJobRuns([/* ... */]); $promise = $client->getJobRunsAsync([/* ... */]);
Retrieves metadata for all runs of a given job definition.
Parameter Syntax
$result = $client->getJobRuns([
'JobName' => '<string>', // REQUIRED
'MaxResults' => <integer>,
'NextToken' => '<string>',
]);
Parameter Details
Members
Result Syntax
[
'JobRuns' => [
[
'AllocatedCapacity' => <integer>,
'Arguments' => ['<string>', ...],
'Attempt' => <integer>,
'CompletedOn' => <DateTime>,
'ErrorMessage' => '<string>',
'ExecutionTime' => <integer>,
'GlueVersion' => '<string>',
'Id' => '<string>',
'JobName' => '<string>',
'JobRunState' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
'LastModifiedOn' => <DateTime>,
'LogGroupName' => '<string>',
'MaxCapacity' => <float>,
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'NumberOfWorkers' => <integer>,
'PredecessorRuns' => [
[
'JobName' => '<string>',
'RunId' => '<string>',
],
// ...
],
'PreviousRunId' => '<string>',
'SecurityConfiguration' => '<string>',
'StartedOn' => <DateTime>,
'Timeout' => <integer>,
'TriggerName' => '<string>',
'WorkerType' => 'Standard|G.1X|G.2X',
],
// ...
],
'NextToken' => '<string>',
]
Result Details
Members
- JobRuns
-
- Type: Array of JobRun structures
A list of job-run metadata objects.
- NextToken
-
- Type: string
A continuation token, if not all requested job runs have been returned.
Errors
-
The input provided was not valid.
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
GetJobs
$result = $client->getJobs([/* ... */]); $promise = $client->getJobsAsync([/* ... */]);
Retrieves all current job definitions.
Parameter Syntax
$result = $client->getJobs([
'MaxResults' => <integer>,
'NextToken' => '<string>',
]);
Parameter Details
Members
Result Syntax
[
'Jobs' => [
[
'AllocatedCapacity' => <integer>,
'Command' => [
'Name' => '<string>',
'PythonVersion' => '<string>',
'ScriptLocation' => '<string>',
],
'Connections' => [
'Connections' => ['<string>', ...],
],
'CreatedOn' => <DateTime>,
'DefaultArguments' => ['<string>', ...],
'Description' => '<string>',
'ExecutionProperty' => [
'MaxConcurrentRuns' => <integer>,
],
'GlueVersion' => '<string>',
'LastModifiedOn' => <DateTime>,
'LogUri' => '<string>',
'MaxCapacity' => <float>,
'MaxRetries' => <integer>,
'Name' => '<string>',
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'NumberOfWorkers' => <integer>,
'Role' => '<string>',
'SecurityConfiguration' => '<string>',
'Timeout' => <integer>,
'WorkerType' => 'Standard|G.1X|G.2X',
],
// ...
],
'NextToken' => '<string>',
]
Result Details
Members
- Jobs
-
- Type: Array of Job structures
A list of job definitions.
- NextToken
-
- Type: string
A continuation token, if not all job definitions have yet been returned.
Errors
-
The input provided was not valid.
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
GetMLTaskRun
$result = $client->getMLTaskRun([/* ... */]); $promise = $client->getMLTaskRunAsync([/* ... */]);
Gets details for a specific task run on a machine learning transform. Machine learning task runs are asynchronous tasks that AWS Glue runs on your behalf as part of various machine learning workflows. You can check the stats of any task run by calling GetMLTaskRun with the TaskRunID and its parent transform's TransformID.
Parameter Syntax
$result = $client->getMLTaskRun([
'TaskRunId' => '<string>', // REQUIRED
'TransformId' => '<string>', // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'CompletedOn' => <DateTime>,
'ErrorString' => '<string>',
'ExecutionTime' => <integer>,
'LastModifiedOn' => <DateTime>,
'LogGroupName' => '<string>',
'Properties' => [
'ExportLabelsTaskRunProperties' => [
'OutputS3Path' => '<string>',
],
'FindMatchesTaskRunProperties' => [
'JobId' => '<string>',
'JobName' => '<string>',
'JobRunId' => '<string>',
],
'ImportLabelsTaskRunProperties' => [
'InputS3Path' => '<string>',
'Replace' => true || false,
],
'LabelingSetGenerationTaskRunProperties' => [
'OutputS3Path' => '<string>',
],
'TaskType' => 'EVALUATION|LABELING_SET_GENERATION|IMPORT_LABELS|EXPORT_LABELS|FIND_MATCHES',
],
'StartedOn' => <DateTime>,
'Status' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
'TaskRunId' => '<string>',
'TransformId' => '<string>',
]
Result Details
Members
- CompletedOn
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The date and time when this task run was completed.
- ErrorString
-
- Type: string
The error strings that are associated with the task run.
- ExecutionTime
-
- Type: int
The amount of time (in seconds) that the task run consumed resources.
- LastModifiedOn
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The date and time when this task run was last modified.
- LogGroupName
-
- Type: string
The names of the log groups that are associated with the task run.
- Properties
-
- Type: TaskRunProperties structure
The list of properties that are associated with the task run.
- StartedOn
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The date and time when this task run started.
- Status
-
- Type: string
The status for this task run.
- TaskRunId
-
- Type: string
The unique run identifier associated with this run.
- TransformId
-
- Type: string
The unique identifier of the task run.
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
The operation timed out.
-
An internal service error occurred.
GetMLTaskRuns
$result = $client->getMLTaskRuns([/* ... */]); $promise = $client->getMLTaskRunsAsync([/* ... */]);
Gets a list of runs for a machine learning transform. Machine learning task runs are asynchronous tasks that AWS Glue runs on your behalf as part of various machine learning workflows. You can get a sortable, filterable list of machine learning task runs by calling GetMLTaskRuns with their parent transform's TransformID and other optional parameters as documented in this section.
This operation returns a list of historic runs and must be paginated.
Parameter Syntax
$result = $client->getMLTaskRuns([
'Filter' => [
'StartedAfter' => <integer || string || DateTime>,
'StartedBefore' => <integer || string || DateTime>,
'Status' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
'TaskRunType' => 'EVALUATION|LABELING_SET_GENERATION|IMPORT_LABELS|EXPORT_LABELS|FIND_MATCHES',
],
'MaxResults' => <integer>,
'NextToken' => '<string>',
'Sort' => [
'Column' => 'TASK_RUN_TYPE|STATUS|STARTED', // REQUIRED
'SortDirection' => 'DESCENDING|ASCENDING', // REQUIRED
],
'TransformId' => '<string>', // REQUIRED
]);
Parameter Details
Members
- Filter
-
- Type: TaskRunFilterCriteria structure
The filter criteria, in the
TaskRunFilterCriteriastructure, for the task run. - MaxResults
-
- Type: int
The maximum number of results to return.
- NextToken
-
- Type: string
A token for pagination of the results. The default is empty.
- Sort
-
- Type: TaskRunSortCriteria structure
The sorting criteria, in the
TaskRunSortCriteriastructure, for the task run. - TransformId
-
- Required: Yes
- Type: string
The unique identifier of the machine learning transform.
Result Syntax
[
'NextToken' => '<string>',
'TaskRuns' => [
[
'CompletedOn' => <DateTime>,
'ErrorString' => '<string>',
'ExecutionTime' => <integer>,
'LastModifiedOn' => <DateTime>,
'LogGroupName' => '<string>',
'Properties' => [
'ExportLabelsTaskRunProperties' => [
'OutputS3Path' => '<string>',
],
'FindMatchesTaskRunProperties' => [
'JobId' => '<string>',
'JobName' => '<string>',
'JobRunId' => '<string>',
],
'ImportLabelsTaskRunProperties' => [
'InputS3Path' => '<string>',
'Replace' => true || false,
],
'LabelingSetGenerationTaskRunProperties' => [
'OutputS3Path' => '<string>',
],
'TaskType' => 'EVALUATION|LABELING_SET_GENERATION|IMPORT_LABELS|EXPORT_LABELS|FIND_MATCHES',
],
'StartedOn' => <DateTime>,
'Status' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
'TaskRunId' => '<string>',
'TransformId' => '<string>',
],
// ...
],
]
Result Details
Members
- NextToken
-
- Type: string
A pagination token, if more results are available.
- TaskRuns
-
- Type: Array of TaskRun structures
A list of task runs that are associated with the transform.
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
The operation timed out.
-
An internal service error occurred.
GetMLTransform
$result = $client->getMLTransform([/* ... */]); $promise = $client->getMLTransformAsync([/* ... */]);
Gets an AWS Glue machine learning transform artifact and all its corresponding metadata. Machine learning transforms are a special type of transform that use machine learning to learn the details of the transformation to be performed by learning from examples provided by humans. These transformations are then saved by AWS Glue. You can retrieve their metadata by calling GetMLTransform.
Parameter Syntax
$result = $client->getMLTransform([
'TransformId' => '<string>', // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'CreatedOn' => <DateTime>,
'Description' => '<string>',
'EvaluationMetrics' => [
'FindMatchesMetrics' => [
'AreaUnderPRCurve' => <float>,
'ConfusionMatrix' => [
'NumFalseNegatives' => <integer>,
'NumFalsePositives' => <integer>,
'NumTrueNegatives' => <integer>,
'NumTruePositives' => <integer>,
],
'F1' => <float>,
'Precision' => <float>,
'Recall' => <float>,
],
'TransformType' => 'FIND_MATCHES',
],
'InputRecordTables' => [
[
'CatalogId' => '<string>',
'ConnectionName' => '<string>',
'DatabaseName' => '<string>',
'TableName' => '<string>',
],
// ...
],
'LabelCount' => <integer>,
'LastModifiedOn' => <DateTime>,
'MaxCapacity' => <float>,
'MaxRetries' => <integer>,
'Name' => '<string>',
'NumberOfWorkers' => <integer>,
'Parameters' => [
'FindMatchesParameters' => [
'AccuracyCostTradeoff' => <float>,
'EnforceProvidedLabels' => true || false,
'PrecisionRecallTradeoff' => <float>,
'PrimaryKeyColumnName' => '<string>',
],
'TransformType' => 'FIND_MATCHES',
],
'Role' => '<string>',
'Schema' => [
[
'DataType' => '<string>',
'Name' => '<string>',
],
// ...
],
'Status' => 'NOT_READY|READY|DELETING',
'Timeout' => <integer>,
'TransformId' => '<string>',
'WorkerType' => 'Standard|G.1X|G.2X',
]
Result Details
Members
- CreatedOn
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The date and time when the transform was created.
- Description
-
- Type: string
A description of the transform.
- EvaluationMetrics
-
- Type: EvaluationMetrics structure
The latest evaluation metrics.
- InputRecordTables
-
- Type: Array of GlueTable structures
A list of AWS Glue table definitions used by the transform.
- LabelCount
-
- Type: int
The number of labels available for this transform.
- LastModifiedOn
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The date and time when the transform was last modified.
- MaxCapacity
-
- Type: double
The number of AWS Glue data processing units (DPUs) that are allocated to task runs for this transform. You can allocate from 2 to 100 DPUs; the default is 10. A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB of memory. For more information, see the AWS Glue pricing page.
When the
WorkerTypefield is set to a value other thanStandard, theMaxCapacityfield is set automatically and becomes read-only. - MaxRetries
-
- Type: int
The maximum number of times to retry a task for this transform after a task run fails.
- Name
-
- Type: string
The unique name given to the transform when it was created.
- NumberOfWorkers
-
- Type: int
The number of workers of a defined
workerTypethat are allocated when this task runs. - Parameters
-
- Type: TransformParameters structure
The configuration parameters that are specific to the algorithm used.
- Role
-
- Type: string
The name or Amazon Resource Name (ARN) of the IAM role with the required permissions.
- Schema
-
- Type: Array of SchemaColumn structures
The
Map<Column, Type>object that represents the schema that this transform accepts. Has an upper bound of 100 columns. - Status
-
- Type: string
The last known status of the transform (to indicate whether it can be used or not). One of "NOT_READY", "READY", or "DELETING".
- Timeout
-
- Type: int
The timeout for a task run for this transform in minutes. This is the maximum time that a task run for this transform can consume resources before it is terminated and enters
TIMEOUTstatus. The default is 2,880 minutes (48 hours). - TransformId
-
- Type: string
The unique identifier of the transform, generated at the time that the transform was created.
- WorkerType
-
- Type: string
The type of predefined worker that is allocated when this task runs. Accepts a value of Standard, G.1X, or G.2X.
-
For the
Standardworker type, each worker provides 4 vCPU, 16 GB of memory and a 50GB disk, and 2 executors per worker. -
For the
G.1Xworker type, each worker provides 4 vCPU, 16 GB of memory and a 64GB disk, and 1 executor per worker. -
For the
G.2Xworker type, each worker provides 8 vCPU, 32 GB of memory and a 128GB disk, and 1 executor per worker.
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
The operation timed out.
-
An internal service error occurred.
GetMLTransforms
$result = $client->getMLTransforms([/* ... */]); $promise = $client->getMLTransformsAsync([/* ... */]);
Gets a sortable, filterable list of existing AWS Glue machine learning transforms. Machine learning transforms are a special type of transform that use machine learning to learn the details of the transformation to be performed by learning from examples provided by humans. These transformations are then saved by AWS Glue, and you can retrieve their metadata by calling GetMLTransforms.
Parameter Syntax
$result = $client->getMLTransforms([
'Filter' => [
'CreatedAfter' => <integer || string || DateTime>,
'CreatedBefore' => <integer || string || DateTime>,
'LastModifiedAfter' => <integer || string || DateTime>,
'LastModifiedBefore' => <integer || string || DateTime>,
'Name' => '<string>',
'Schema' => [
[
'DataType' => '<string>',
'Name' => '<string>',
],
// ...
],
'Status' => 'NOT_READY|READY|DELETING',
'TransformType' => 'FIND_MATCHES',
],
'MaxResults' => <integer>,
'NextToken' => '<string>',
'Sort' => [
'Column' => 'NAME|TRANSFORM_TYPE|STATUS|CREATED|LAST_MODIFIED', // REQUIRED
'SortDirection' => 'DESCENDING|ASCENDING', // REQUIRED
],
]);
Parameter Details
Members
- Filter
-
- Type: TransformFilterCriteria structure
The filter transformation criteria.
- MaxResults
-
- Type: int
The maximum number of results to return.
- NextToken
-
- Type: string
A paginated token to offset the results.
- Sort
-
- Type: TransformSortCriteria structure
The sorting criteria.
Result Syntax
[
'NextToken' => '<string>',
'Transforms' => [
[
'CreatedOn' => <DateTime>,
'Description' => '<string>',
'EvaluationMetrics' => [
'FindMatchesMetrics' => [
'AreaUnderPRCurve' => <float>,
'ConfusionMatrix' => [
'NumFalseNegatives' => <integer>,
'NumFalsePositives' => <integer>,
'NumTrueNegatives' => <integer>,
'NumTruePositives' => <integer>,
],
'F1' => <float>,
'Precision' => <float>,
'Recall' => <float>,
],
'TransformType' => 'FIND_MATCHES',
],
'InputRecordTables' => [
[
'CatalogId' => '<string>',
'ConnectionName' => '<string>',
'DatabaseName' => '<string>',
'TableName' => '<string>',
],
// ...
],
'LabelCount' => <integer>,
'LastModifiedOn' => <DateTime>,
'MaxCapacity' => <float>,
'MaxRetries' => <integer>,
'Name' => '<string>',
'NumberOfWorkers' => <integer>,
'Parameters' => [
'FindMatchesParameters' => [
'AccuracyCostTradeoff' => <float>,
'EnforceProvidedLabels' => true || false,
'PrecisionRecallTradeoff' => <float>,
'PrimaryKeyColumnName' => '<string>',
],
'TransformType' => 'FIND_MATCHES',
],
'Role' => '<string>',
'Schema' => [
[
'DataType' => '<string>',
'Name' => '<string>',
],
// ...
],
'Status' => 'NOT_READY|READY|DELETING',
'Timeout' => <integer>,
'TransformId' => '<string>',
'WorkerType' => 'Standard|G.1X|G.2X',
],
// ...
],
]
Result Details
Members
- NextToken
-
- Type: string
A pagination token, if more results are available.
- Transforms
-
- Required: Yes
- Type: Array of MLTransform structures
A list of machine learning transforms.
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
The operation timed out.
-
An internal service error occurred.
GetMapping
$result = $client->getMapping([/* ... */]); $promise = $client->getMappingAsync([/* ... */]);
Creates mappings.
Parameter Syntax
$result = $client->getMapping([
'Location' => [
'DynamoDB' => [
[
'Name' => '<string>', // REQUIRED
'Param' => true || false,
'Value' => '<string>', // REQUIRED
],
// ...
],
'Jdbc' => [
[
'Name' => '<string>', // REQUIRED
'Param' => true || false,
'Value' => '<string>', // REQUIRED
],
// ...
],
'S3' => [
[
'Name' => '<string>', // REQUIRED
'Param' => true || false,
'Value' => '<string>', // REQUIRED
],
// ...
],
],
'Sinks' => [
[
'DatabaseName' => '<string>', // REQUIRED
'TableName' => '<string>', // REQUIRED
],
// ...
],
'Source' => [ // REQUIRED
'DatabaseName' => '<string>', // REQUIRED
'TableName' => '<string>', // REQUIRED
],
]);
Parameter Details
Members
- Location
-
- Type: Location structure
Parameters for the mapping.
- Sinks
-
- Type: Array of CatalogEntry structures
A list of target tables.
- Source
-
- Required: Yes
- Type: CatalogEntry structure
Specifies the source table.
Result Syntax
[
'Mapping' => [
[
'SourcePath' => '<string>',
'SourceTable' => '<string>',
'SourceType' => '<string>',
'TargetPath' => '<string>',
'TargetTable' => '<string>',
'TargetType' => '<string>',
],
// ...
],
]
Result Details
Members
- Mapping
-
- Required: Yes
- Type: Array of MappingEntry structures
A list of mappings to the specified targets.
Errors
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
-
A specified entity does not exist
GetPartition
$result = $client->getPartition([/* ... */]); $promise = $client->getPartitionAsync([/* ... */]);
Retrieves information about a specified partition.
Parameter Syntax
$result = $client->getPartition([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'PartitionValues' => ['<string>', ...], // REQUIRED
'TableName' => '<string>', // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog where the partition in question resides. If none is provided, the AWS account ID is used by default.
- DatabaseName
-
- Required: Yes
- Type: string
The name of the catalog database where the partition resides.
- PartitionValues
-
- Required: Yes
- Type: Array of strings
The values that define the partition.
- TableName
-
- Required: Yes
- Type: string
The name of the partition's table.
Result Syntax
[
'Partition' => [
'CreationTime' => <DateTime>,
'DatabaseName' => '<string>',
'LastAccessTime' => <DateTime>,
'LastAnalyzedTime' => <DateTime>,
'Parameters' => ['<string>', ...],
'StorageDescriptor' => [
'BucketColumns' => ['<string>', ...],
'Columns' => [
[
'Comment' => '<string>',
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'Type' => '<string>',
],
// ...
],
'Compressed' => true || false,
'InputFormat' => '<string>',
'Location' => '<string>',
'NumberOfBuckets' => <integer>,
'OutputFormat' => '<string>',
'Parameters' => ['<string>', ...],
'SerdeInfo' => [
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'SerializationLibrary' => '<string>',
],
'SkewedInfo' => [
'SkewedColumnNames' => ['<string>', ...],
'SkewedColumnValueLocationMaps' => ['<string>', ...],
'SkewedColumnValues' => ['<string>', ...],
],
'SortColumns' => [
[
'Column' => '<string>',
'SortOrder' => <integer>,
],
// ...
],
'StoredAsSubDirectories' => true || false,
],
'TableName' => '<string>',
'Values' => ['<string>', ...],
],
]
Result Details
Members
- Partition
-
- Type: Partition structure
The requested information, in the form of a
Partitionobject.
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
-
An encryption operation failed.
GetPartitions
$result = $client->getPartitions([/* ... */]); $promise = $client->getPartitionsAsync([/* ... */]);
Retrieves information about the partitions in a table.
Parameter Syntax
$result = $client->getPartitions([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'Expression' => '<string>',
'MaxResults' => <integer>,
'NextToken' => '<string>',
'Segment' => [
'SegmentNumber' => <integer>, // REQUIRED
'TotalSegments' => <integer>, // REQUIRED
],
'TableName' => '<string>', // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog where the partitions in question reside. If none is provided, the AWS account ID is used by default.
- DatabaseName
-
- Required: Yes
- Type: string
The name of the catalog database where the partitions reside.
- Expression
-
- Type: string
An expression that filters the partitions to be returned.
The expression uses SQL syntax similar to the SQL
WHEREfilter clause. The SQL statement parser JSQLParser parses the expression.Operators: The following are the operators that you can use in the
ExpressionAPI call:- =
-
Checks whether the values of the two operands are equal; if yes, then the condition becomes true.
Example: Assume 'variable a' holds 10 and 'variable b' holds 20.
(a = b) is not true.
- < >
-
Checks whether the values of two operands are equal; if the values are not equal, then the condition becomes true.
Example: (a < > b) is true.
- >
-
Checks whether the value of the left operand is greater than the value of the right operand; if yes, then the condition becomes true.
Example: (a > b) is not true.
- <
-
Checks whether the value of the left operand is less than the value of the right operand; if yes, then the condition becomes true.
Example: (a < b) is true.
- >=
-
Checks whether the value of the left operand is greater than or equal to the value of the right operand; if yes, then the condition becomes true.
Example: (a >= b) is not true.
- <=
-
Checks whether the value of the left operand is less than or equal to the value of the right operand; if yes, then the condition becomes true.
Example: (a <= b) is true.
- AND, OR, IN, BETWEEN, LIKE, NOT, IS NULL
-
Logical operators.
Supported Partition Key Types: The following are the supported partition keys.
-
string -
date -
timestamp -
int -
bigint -
long -
tinyint -
smallint -
decimal
If an invalid type is encountered, an exception is thrown.
The following list shows the valid operators on each type. When you define a crawler, the
partitionKeytype is created as aSTRING, to be compatible with the catalog partitions.Sample API Call:
- MaxResults
-
- Type: int
The maximum number of partitions to return in a single response.
- NextToken
-
- Type: string
A continuation token, if this is not the first call to retrieve these partitions.
- Segment
-
- Type: Segment structure
The segment of the table's partitions to scan in this request.
- TableName
-
- Required: Yes
- Type: string
The name of the partitions' table.
Result Syntax
[
'NextToken' => '<string>',
'Partitions' => [
[
'CreationTime' => <DateTime>,
'DatabaseName' => '<string>',
'LastAccessTime' => <DateTime>,
'LastAnalyzedTime' => <DateTime>,
'Parameters' => ['<string>', ...],
'StorageDescriptor' => [
'BucketColumns' => ['<string>', ...],
'Columns' => [
[
'Comment' => '<string>',
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'Type' => '<string>',
],
// ...
],
'Compressed' => true || false,
'InputFormat' => '<string>',
'Location' => '<string>',
'NumberOfBuckets' => <integer>,
'OutputFormat' => '<string>',
'Parameters' => ['<string>', ...],
'SerdeInfo' => [
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'SerializationLibrary' => '<string>',
],
'SkewedInfo' => [
'SkewedColumnNames' => ['<string>', ...],
'SkewedColumnValueLocationMaps' => ['<string>', ...],
'SkewedColumnValues' => ['<string>', ...],
],
'SortColumns' => [
[
'Column' => '<string>',
'SortOrder' => <integer>,
],
// ...
],
'StoredAsSubDirectories' => true || false,
],
'TableName' => '<string>',
'Values' => ['<string>', ...],
],
// ...
],
]
Result Details
Members
- NextToken
-
- Type: string
A continuation token, if the returned list of partitions does not include the last one.
- Partitions
-
- Type: Array of Partition structures
A list of requested partitions.
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
The operation timed out.
-
An internal service error occurred.
-
An encryption operation failed.
GetPlan
$result = $client->getPlan([/* ... */]); $promise = $client->getPlanAsync([/* ... */]);
Gets code to perform a specified mapping.
Parameter Syntax
$result = $client->getPlan([
'Language' => 'PYTHON|SCALA',
'Location' => [
'DynamoDB' => [
[
'Name' => '<string>', // REQUIRED
'Param' => true || false,
'Value' => '<string>', // REQUIRED
],
// ...
],
'Jdbc' => [
[
'Name' => '<string>', // REQUIRED
'Param' => true || false,
'Value' => '<string>', // REQUIRED
],
// ...
],
'S3' => [
[
'Name' => '<string>', // REQUIRED
'Param' => true || false,
'Value' => '<string>', // REQUIRED
],
// ...
],
],
'Mapping' => [ // REQUIRED
[
'SourcePath' => '<string>',
'SourceTable' => '<string>',
'SourceType' => '<string>',
'TargetPath' => '<string>',
'TargetTable' => '<string>',
'TargetType' => '<string>',
],
// ...
],
'Sinks' => [
[
'DatabaseName' => '<string>', // REQUIRED
'TableName' => '<string>', // REQUIRED
],
// ...
],
'Source' => [ // REQUIRED
'DatabaseName' => '<string>', // REQUIRED
'TableName' => '<string>', // REQUIRED
],
]);
Parameter Details
Members
- Language
-
- Type: string
The programming language of the code to perform the mapping.
- Location
-
- Type: Location structure
The parameters for the mapping.
- Mapping
-
- Required: Yes
- Type: Array of MappingEntry structures
The list of mappings from a source table to target tables.
- Sinks
-
- Type: Array of CatalogEntry structures
The target tables.
- Source
-
- Required: Yes
- Type: CatalogEntry structure
The source table.
Result Syntax
[
'PythonScript' => '<string>',
'ScalaCode' => '<string>',
]
Result Details
Members
Errors
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
GetResourcePolicy
$result = $client->getResourcePolicy([/* ... */]); $promise = $client->getResourcePolicyAsync([/* ... */]);
Retrieves a specified resource policy.
Parameter Syntax
$result = $client->getResourcePolicy([ ]);
Parameter Details
Members
Result Syntax
[
'CreateTime' => <DateTime>,
'PolicyHash' => '<string>',
'PolicyInJson' => '<string>',
'UpdateTime' => <DateTime>,
]
Result Details
Members
- CreateTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The date and time at which the policy was created.
- PolicyHash
-
- Type: string
Contains the hash value associated with this policy.
- PolicyInJson
-
- Type: string
Contains the requested policy document, in JSON format.
- UpdateTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The date and time at which the policy was last updated.
Errors
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
-
The input provided was not valid.
GetSecurityConfiguration
$result = $client->getSecurityConfiguration([/* ... */]); $promise = $client->getSecurityConfigurationAsync([/* ... */]);
Retrieves a specified security configuration.
Parameter Syntax
$result = $client->getSecurityConfiguration([
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Result Syntax
[
'SecurityConfiguration' => [
'CreatedTimeStamp' => <DateTime>,
'EncryptionConfiguration' => [
'CloudWatchEncryption' => [
'CloudWatchEncryptionMode' => 'DISABLED|SSE-KMS',
'KmsKeyArn' => '<string>',
],
'JobBookmarksEncryption' => [
'JobBookmarksEncryptionMode' => 'DISABLED|CSE-KMS',
'KmsKeyArn' => '<string>',
],
'S3Encryption' => [
[
'KmsKeyArn' => '<string>',
'S3EncryptionMode' => 'DISABLED|SSE-KMS|SSE-S3',
],
// ...
],
],
'Name' => '<string>',
],
]
Result Details
Members
- SecurityConfiguration
-
- Type: SecurityConfiguration structure
The requested security configuration.
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
GetSecurityConfigurations
$result = $client->getSecurityConfigurations([/* ... */]); $promise = $client->getSecurityConfigurationsAsync([/* ... */]);
Retrieves a list of all security configurations.
Parameter Syntax
$result = $client->getSecurityConfigurations([
'MaxResults' => <integer>,
'NextToken' => '<string>',
]);
Parameter Details
Members
Result Syntax
[
'NextToken' => '<string>',
'SecurityConfigurations' => [
[
'CreatedTimeStamp' => <DateTime>,
'EncryptionConfiguration' => [
'CloudWatchEncryption' => [
'CloudWatchEncryptionMode' => 'DISABLED|SSE-KMS',
'KmsKeyArn' => '<string>',
],
'JobBookmarksEncryption' => [
'JobBookmarksEncryptionMode' => 'DISABLED|CSE-KMS',
'KmsKeyArn' => '<string>',
],
'S3Encryption' => [
[
'KmsKeyArn' => '<string>',
'S3EncryptionMode' => 'DISABLED|SSE-KMS|SSE-S3',
],
// ...
],
],
'Name' => '<string>',
],
// ...
],
]
Result Details
Members
- NextToken
-
- Type: string
A continuation token, if there are more security configurations to return.
- SecurityConfigurations
-
- Type: Array of SecurityConfiguration structures
A list of security configurations.
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
GetTable
$result = $client->getTable([/* ... */]); $promise = $client->getTableAsync([/* ... */]);
Retrieves the Table definition in a Data Catalog for a specified table.
Parameter Syntax
$result = $client->getTable([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog where the table resides. If none is provided, the AWS account ID is used by default.
- DatabaseName
-
- Required: Yes
- Type: string
The name of the database in the catalog in which the table resides. For Hive compatibility, this name is entirely lowercase.
- Name
-
- Required: Yes
- Type: string
The name of the table for which to retrieve the definition. For Hive compatibility, this name is entirely lowercase.
Result Syntax
[
'Table' => [
'CreateTime' => <DateTime>,
'CreatedBy' => '<string>',
'DatabaseName' => '<string>',
'Description' => '<string>',
'IsRegisteredWithLakeFormation' => true || false,
'LastAccessTime' => <DateTime>,
'LastAnalyzedTime' => <DateTime>,
'Name' => '<string>',
'Owner' => '<string>',
'Parameters' => ['<string>', ...],
'PartitionKeys' => [
[
'Comment' => '<string>',
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'Type' => '<string>',
],
// ...
],
'Retention' => <integer>,
'StorageDescriptor' => [
'BucketColumns' => ['<string>', ...],
'Columns' => [
[
'Comment' => '<string>',
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'Type' => '<string>',
],
// ...
],
'Compressed' => true || false,
'InputFormat' => '<string>',
'Location' => '<string>',
'NumberOfBuckets' => <integer>,
'OutputFormat' => '<string>',
'Parameters' => ['<string>', ...],
'SerdeInfo' => [
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'SerializationLibrary' => '<string>',
],
'SkewedInfo' => [
'SkewedColumnNames' => ['<string>', ...],
'SkewedColumnValueLocationMaps' => ['<string>', ...],
'SkewedColumnValues' => ['<string>', ...],
],
'SortColumns' => [
[
'Column' => '<string>',
'SortOrder' => <integer>,
],
// ...
],
'StoredAsSubDirectories' => true || false,
],
'TableType' => '<string>',
'UpdateTime' => <DateTime>,
'ViewExpandedText' => '<string>',
'ViewOriginalText' => '<string>',
],
]
Result Details
Members
- Table
-
- Type: Table structure
The
Tableobject that defines the specified table.
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
-
An encryption operation failed.
GetTableVersion
$result = $client->getTableVersion([/* ... */]); $promise = $client->getTableVersionAsync([/* ... */]);
Retrieves a specified version of a table.
Parameter Syntax
$result = $client->getTableVersion([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'TableName' => '<string>', // REQUIRED
'VersionId' => '<string>',
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog where the tables reside. If none is provided, the AWS account ID is used by default.
- DatabaseName
-
- Required: Yes
- Type: string
The database in the catalog in which the table resides. For Hive compatibility, this name is entirely lowercase.
- TableName
-
- Required: Yes
- Type: string
The name of the table. For Hive compatibility, this name is entirely lowercase.
- VersionId
-
- Type: string
The ID value of the table version to be retrieved. A
VersionIDis a string representation of an integer. Each version is incremented by 1.
Result Syntax
[
'TableVersion' => [
'Table' => [
'CreateTime' => <DateTime>,
'CreatedBy' => '<string>',
'DatabaseName' => '<string>',
'Description' => '<string>',
'IsRegisteredWithLakeFormation' => true || false,
'LastAccessTime' => <DateTime>,
'LastAnalyzedTime' => <DateTime>,
'Name' => '<string>',
'Owner' => '<string>',
'Parameters' => ['<string>', ...],
'PartitionKeys' => [
[
'Comment' => '<string>',
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'Type' => '<string>',
],
// ...
],
'Retention' => <integer>,
'StorageDescriptor' => [
'BucketColumns' => ['<string>', ...],
'Columns' => [
[
'Comment' => '<string>',
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'Type' => '<string>',
],
// ...
],
'Compressed' => true || false,
'InputFormat' => '<string>',
'Location' => '<string>',
'NumberOfBuckets' => <integer>,
'OutputFormat' => '<string>',
'Parameters' => ['<string>', ...],
'SerdeInfo' => [
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'SerializationLibrary' => '<string>',
],
'SkewedInfo' => [
'SkewedColumnNames' => ['<string>', ...],
'SkewedColumnValueLocationMaps' => ['<string>', ...],
'SkewedColumnValues' => ['<string>', ...],
],
'SortColumns' => [
[
'Column' => '<string>',
'SortOrder' => <integer>,
],
// ...
],
'StoredAsSubDirectories' => true || false,
],
'TableType' => '<string>',
'UpdateTime' => <DateTime>,
'ViewExpandedText' => '<string>',
'ViewOriginalText' => '<string>',
],
'VersionId' => '<string>',
],
]
Result Details
Members
- TableVersion
-
- Type: TableVersion structure
The requested table version.
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
-
An encryption operation failed.
GetTableVersions
$result = $client->getTableVersions([/* ... */]); $promise = $client->getTableVersionsAsync([/* ... */]);
Retrieves a list of strings that identify available versions of a specified table.
Parameter Syntax
$result = $client->getTableVersions([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'MaxResults' => <integer>,
'NextToken' => '<string>',
'TableName' => '<string>', // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog where the tables reside. If none is provided, the AWS account ID is used by default.
- DatabaseName
-
- Required: Yes
- Type: string
The database in the catalog in which the table resides. For Hive compatibility, this name is entirely lowercase.
- MaxResults
-
- Type: int
The maximum number of table versions to return in one response.
- NextToken
-
- Type: string
A continuation token, if this is not the first call.
- TableName
-
- Required: Yes
- Type: string
The name of the table. For Hive compatibility, this name is entirely lowercase.
Result Syntax
[
'NextToken' => '<string>',
'TableVersions' => [
[
'Table' => [
'CreateTime' => <DateTime>,
'CreatedBy' => '<string>',
'DatabaseName' => '<string>',
'Description' => '<string>',
'IsRegisteredWithLakeFormation' => true || false,
'LastAccessTime' => <DateTime>,
'LastAnalyzedTime' => <DateTime>,
'Name' => '<string>',
'Owner' => '<string>',
'Parameters' => ['<string>', ...],
'PartitionKeys' => [
[
'Comment' => '<string>',
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'Type' => '<string>',
],
// ...
],
'Retention' => <integer>,
'StorageDescriptor' => [
'BucketColumns' => ['<string>', ...],
'Columns' => [
[
'Comment' => '<string>',
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'Type' => '<string>',
],
// ...
],
'Compressed' => true || false,
'InputFormat' => '<string>',
'Location' => '<string>',
'NumberOfBuckets' => <integer>,
'OutputFormat' => '<string>',
'Parameters' => ['<string>', ...],
'SerdeInfo' => [
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'SerializationLibrary' => '<string>',
],
'SkewedInfo' => [
'SkewedColumnNames' => ['<string>', ...],
'SkewedColumnValueLocationMaps' => ['<string>', ...],
'SkewedColumnValues' => ['<string>', ...],
],
'SortColumns' => [
[
'Column' => '<string>',
'SortOrder' => <integer>,
],
// ...
],
'StoredAsSubDirectories' => true || false,
],
'TableType' => '<string>',
'UpdateTime' => <DateTime>,
'ViewExpandedText' => '<string>',
'ViewOriginalText' => '<string>',
],
'VersionId' => '<string>',
],
// ...
],
]
Result Details
Members
- NextToken
-
- Type: string
A continuation token, if the list of available versions does not include the last one.
- TableVersions
-
- Type: Array of TableVersion structures
A list of strings identifying available versions of the specified table.
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
-
An encryption operation failed.
GetTables
$result = $client->getTables([/* ... */]); $promise = $client->getTablesAsync([/* ... */]);
Retrieves the definitions of some or all of the tables in a given Database.
Parameter Syntax
$result = $client->getTables([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'Expression' => '<string>',
'MaxResults' => <integer>,
'NextToken' => '<string>',
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog where the tables reside. If none is provided, the AWS account ID is used by default.
- DatabaseName
-
- Required: Yes
- Type: string
The database in the catalog whose tables to list. For Hive compatibility, this name is entirely lowercase.
- Expression
-
- Type: string
A regular expression pattern. If present, only those tables whose names match the pattern are returned.
- MaxResults
-
- Type: int
The maximum number of tables to return in a single response.
- NextToken
-
- Type: string
A continuation token, included if this is a continuation call.
Result Syntax
[
'NextToken' => '<string>',
'TableList' => [
[
'CreateTime' => <DateTime>,
'CreatedBy' => '<string>',
'DatabaseName' => '<string>',
'Description' => '<string>',
'IsRegisteredWithLakeFormation' => true || false,
'LastAccessTime' => <DateTime>,
'LastAnalyzedTime' => <DateTime>,
'Name' => '<string>',
'Owner' => '<string>',
'Parameters' => ['<string>', ...],
'PartitionKeys' => [
[
'Comment' => '<string>',
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'Type' => '<string>',
],
// ...
],
'Retention' => <integer>,
'StorageDescriptor' => [
'BucketColumns' => ['<string>', ...],
'Columns' => [
[
'Comment' => '<string>',
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'Type' => '<string>',
],
// ...
],
'Compressed' => true || false,
'InputFormat' => '<string>',
'Location' => '<string>',
'NumberOfBuckets' => <integer>,
'OutputFormat' => '<string>',
'Parameters' => ['<string>', ...],
'SerdeInfo' => [
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'SerializationLibrary' => '<string>',
],
'SkewedInfo' => [
'SkewedColumnNames' => ['<string>', ...],
'SkewedColumnValueLocationMaps' => ['<string>', ...],
'SkewedColumnValues' => ['<string>', ...],
],
'SortColumns' => [
[
'Column' => '<string>',
'SortOrder' => <integer>,
],
// ...
],
'StoredAsSubDirectories' => true || false,
],
'TableType' => '<string>',
'UpdateTime' => <DateTime>,
'ViewExpandedText' => '<string>',
'ViewOriginalText' => '<string>',
],
// ...
],
]
Result Details
Members
- NextToken
-
- Type: string
A continuation token, present if the current list segment is not the last.
- TableList
-
- Type: Array of Table structures
A list of the requested
Tableobjects.
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
The operation timed out.
-
An internal service error occurred.
-
An encryption operation failed.
GetTags
$result = $client->getTags([/* ... */]); $promise = $client->getTagsAsync([/* ... */]);
Retrieves a list of tags associated with a resource.
Parameter Syntax
$result = $client->getTags([
'ResourceArn' => '<string>', // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'Tags' => ['<string>', ...],
]
Result Details
Errors
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
-
A specified entity does not exist
GetTrigger
$result = $client->getTrigger([/* ... */]); $promise = $client->getTriggerAsync([/* ... */]);
Retrieves the definition of a trigger.
Parameter Syntax
$result = $client->getTrigger([
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Result Syntax
[
'Trigger' => [
'Actions' => [
[
'Arguments' => ['<string>', ...],
'CrawlerName' => '<string>',
'JobName' => '<string>',
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'SecurityConfiguration' => '<string>',
'Timeout' => <integer>,
],
// ...
],
'Description' => '<string>',
'Id' => '<string>',
'Name' => '<string>',
'Predicate' => [
'Conditions' => [
[
'CrawlState' => 'RUNNING|SUCCEEDED|CANCELLED|FAILED',
'CrawlerName' => '<string>',
'JobName' => '<string>',
'LogicalOperator' => 'EQUALS',
'State' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
],
// ...
],
'Logical' => 'AND|ANY',
],
'Schedule' => '<string>',
'State' => 'CREATING|CREATED|ACTIVATING|ACTIVATED|DEACTIVATING|DEACTIVATED|DELETING|UPDATING',
'Type' => 'SCHEDULED|CONDITIONAL|ON_DEMAND',
'WorkflowName' => '<string>',
],
]
Result Details
Members
- Trigger
-
- Type: Trigger structure
The requested trigger definition.
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
GetTriggers
$result = $client->getTriggers([/* ... */]); $promise = $client->getTriggersAsync([/* ... */]);
Gets all the triggers associated with a job.
Parameter Syntax
$result = $client->getTriggers([
'DependentJobName' => '<string>',
'MaxResults' => <integer>,
'NextToken' => '<string>',
]);
Parameter Details
Members
- DependentJobName
-
- Type: string
The name of the job to retrieve triggers for. The trigger that can start this job is returned, and if there is no such trigger, all triggers are returned.
- MaxResults
-
- Type: int
The maximum size of the response.
- NextToken
-
- Type: string
A continuation token, if this is a continuation call.
Result Syntax
[
'NextToken' => '<string>',
'Triggers' => [
[
'Actions' => [
[
'Arguments' => ['<string>', ...],
'CrawlerName' => '<string>',
'JobName' => '<string>',
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'SecurityConfiguration' => '<string>',
'Timeout' => <integer>,
],
// ...
],
'Description' => '<string>',
'Id' => '<string>',
'Name' => '<string>',
'Predicate' => [
'Conditions' => [
[
'CrawlState' => 'RUNNING|SUCCEEDED|CANCELLED|FAILED',
'CrawlerName' => '<string>',
'JobName' => '<string>',
'LogicalOperator' => 'EQUALS',
'State' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
],
// ...
],
'Logical' => 'AND|ANY',
],
'Schedule' => '<string>',
'State' => 'CREATING|CREATED|ACTIVATING|ACTIVATED|DEACTIVATING|DEACTIVATED|DELETING|UPDATING',
'Type' => 'SCHEDULED|CONDITIONAL|ON_DEMAND',
'WorkflowName' => '<string>',
],
// ...
],
]
Result Details
Members
- NextToken
-
- Type: string
A continuation token, if not all the requested triggers have yet been returned.
- Triggers
-
- Type: Array of Trigger structures
A list of triggers for the specified job.
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
GetUserDefinedFunction
$result = $client->getUserDefinedFunction([/* ... */]); $promise = $client->getUserDefinedFunctionAsync([/* ... */]);
Retrieves a specified function definition from the Data Catalog.
Parameter Syntax
$result = $client->getUserDefinedFunction([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'FunctionName' => '<string>', // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog where the function to be retrieved is located. If none is provided, the AWS account ID is used by default.
- DatabaseName
-
- Required: Yes
- Type: string
The name of the catalog database where the function is located.
- FunctionName
-
- Required: Yes
- Type: string
The name of the function.
Result Syntax
[
'UserDefinedFunction' => [
'ClassName' => '<string>',
'CreateTime' => <DateTime>,
'FunctionName' => '<string>',
'OwnerName' => '<string>',
'OwnerType' => 'USER|ROLE|GROUP',
'ResourceUris' => [
[
'ResourceType' => 'JAR|FILE|ARCHIVE',
'Uri' => '<string>',
],
// ...
],
],
]
Result Details
Members
- UserDefinedFunction
-
- Type: UserDefinedFunction structure
The requested function definition.
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
-
An encryption operation failed.
GetUserDefinedFunctions
$result = $client->getUserDefinedFunctions([/* ... */]); $promise = $client->getUserDefinedFunctionsAsync([/* ... */]);
Retrieves multiple function definitions from the Data Catalog.
Parameter Syntax
$result = $client->getUserDefinedFunctions([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'MaxResults' => <integer>,
'NextToken' => '<string>',
'Pattern' => '<string>', // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog where the functions to be retrieved are located. If none is provided, the AWS account ID is used by default.
- DatabaseName
-
- Required: Yes
- Type: string
The name of the catalog database where the functions are located.
- MaxResults
-
- Type: int
The maximum number of functions to return in one response.
- NextToken
-
- Type: string
A continuation token, if this is a continuation call.
- Pattern
-
- Required: Yes
- Type: string
An optional function-name pattern string that filters the function definitions returned.
Result Syntax
[
'NextToken' => '<string>',
'UserDefinedFunctions' => [
[
'ClassName' => '<string>',
'CreateTime' => <DateTime>,
'FunctionName' => '<string>',
'OwnerName' => '<string>',
'OwnerType' => 'USER|ROLE|GROUP',
'ResourceUris' => [
[
'ResourceType' => 'JAR|FILE|ARCHIVE',
'Uri' => '<string>',
],
// ...
],
],
// ...
],
]
Result Details
Members
- NextToken
-
- Type: string
A continuation token, if the list of functions returned does not include the last requested function.
- UserDefinedFunctions
-
- Type: Array of UserDefinedFunction structures
A list of requested function definitions.
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
The operation timed out.
-
An internal service error occurred.
-
An encryption operation failed.
GetWorkflow
$result = $client->getWorkflow([/* ... */]); $promise = $client->getWorkflowAsync([/* ... */]);
Retrieves resource metadata for a workflow.
Parameter Syntax
$result = $client->getWorkflow([
'IncludeGraph' => true || false,
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'Workflow' => [
'CreatedOn' => <DateTime>,
'DefaultRunProperties' => ['<string>', ...],
'Description' => '<string>',
'Graph' => [
'Edges' => [
[
'DestinationId' => '<string>',
'SourceId' => '<string>',
],
// ...
],
'Nodes' => [
[
'CrawlerDetails' => [
'Crawls' => [
[
'CompletedOn' => <DateTime>,
'ErrorMessage' => '<string>',
'LogGroup' => '<string>',
'LogStream' => '<string>',
'StartedOn' => <DateTime>,
'State' => 'RUNNING|SUCCEEDED|CANCELLED|FAILED',
],
// ...
],
],
'JobDetails' => [
'JobRuns' => [
[
'AllocatedCapacity' => <integer>,
'Arguments' => ['<string>', ...],
'Attempt' => <integer>,
'CompletedOn' => <DateTime>,
'ErrorMessage' => '<string>',
'ExecutionTime' => <integer>,
'GlueVersion' => '<string>',
'Id' => '<string>',
'JobName' => '<string>',
'JobRunState' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
'LastModifiedOn' => <DateTime>,
'LogGroupName' => '<string>',
'MaxCapacity' => <float>,
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'NumberOfWorkers' => <integer>,
'PredecessorRuns' => [
[
'JobName' => '<string>',
'RunId' => '<string>',
],
// ...
],
'PreviousRunId' => '<string>',
'SecurityConfiguration' => '<string>',
'StartedOn' => <DateTime>,
'Timeout' => <integer>,
'TriggerName' => '<string>',
'WorkerType' => 'Standard|G.1X|G.2X',
],
// ...
],
],
'Name' => '<string>',
'TriggerDetails' => [
'Trigger' => [
'Actions' => [
[
'Arguments' => ['<string>', ...],
'CrawlerName' => '<string>',
'JobName' => '<string>',
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'SecurityConfiguration' => '<string>',
'Timeout' => <integer>,
],
// ...
],
'Description' => '<string>',
'Id' => '<string>',
'Name' => '<string>',
'Predicate' => [
'Conditions' => [
[
'CrawlState' => 'RUNNING|SUCCEEDED|CANCELLED|FAILED',
'CrawlerName' => '<string>',
'JobName' => '<string>',
'LogicalOperator' => 'EQUALS',
'State' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
],
// ...
],
'Logical' => 'AND|ANY',
],
'Schedule' => '<string>',
'State' => 'CREATING|CREATED|ACTIVATING|ACTIVATED|DEACTIVATING|DEACTIVATED|DELETING|UPDATING',
'Type' => 'SCHEDULED|CONDITIONAL|ON_DEMAND',
'WorkflowName' => '<string>',
],
],
'Type' => 'CRAWLER|JOB|TRIGGER',
'UniqueId' => '<string>',
],
// ...
],
],
'LastModifiedOn' => <DateTime>,
'LastRun' => [
'CompletedOn' => <DateTime>,
'Graph' => [
'Edges' => [
[
'DestinationId' => '<string>',
'SourceId' => '<string>',
],
// ...
],
'Nodes' => [
[
'CrawlerDetails' => [
'Crawls' => [
[
'CompletedOn' => <DateTime>,
'ErrorMessage' => '<string>',
'LogGroup' => '<string>',
'LogStream' => '<string>',
'StartedOn' => <DateTime>,
'State' => 'RUNNING|SUCCEEDED|CANCELLED|FAILED',
],
// ...
],
],
'JobDetails' => [
'JobRuns' => [
[
'AllocatedCapacity' => <integer>,
'Arguments' => ['<string>', ...],
'Attempt' => <integer>,
'CompletedOn' => <DateTime>,
'ErrorMessage' => '<string>',
'ExecutionTime' => <integer>,
'GlueVersion' => '<string>',
'Id' => '<string>',
'JobName' => '<string>',
'JobRunState' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
'LastModifiedOn' => <DateTime>,
'LogGroupName' => '<string>',
'MaxCapacity' => <float>,
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'NumberOfWorkers' => <integer>,
'PredecessorRuns' => [
[
'JobName' => '<string>',
'RunId' => '<string>',
],
// ...
],
'PreviousRunId' => '<string>',
'SecurityConfiguration' => '<string>',
'StartedOn' => <DateTime>,
'Timeout' => <integer>,
'TriggerName' => '<string>',
'WorkerType' => 'Standard|G.1X|G.2X',
],
// ...
],
],
'Name' => '<string>',
'TriggerDetails' => [
'Trigger' => [
'Actions' => [
[
'Arguments' => ['<string>', ...],
'CrawlerName' => '<string>',
'JobName' => '<string>',
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'SecurityConfiguration' => '<string>',
'Timeout' => <integer>,
],
// ...
],
'Description' => '<string>',
'Id' => '<string>',
'Name' => '<string>',
'Predicate' => [
'Conditions' => [
[
'CrawlState' => 'RUNNING|SUCCEEDED|CANCELLED|FAILED',
'CrawlerName' => '<string>',
'JobName' => '<string>',
'LogicalOperator' => 'EQUALS',
'State' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
],
// ...
],
'Logical' => 'AND|ANY',
],
'Schedule' => '<string>',
'State' => 'CREATING|CREATED|ACTIVATING|ACTIVATED|DEACTIVATING|DEACTIVATED|DELETING|UPDATING',
'Type' => 'SCHEDULED|CONDITIONAL|ON_DEMAND',
'WorkflowName' => '<string>',
],
],
'Type' => 'CRAWLER|JOB|TRIGGER',
'UniqueId' => '<string>',
],
// ...
],
],
'Name' => '<string>',
'StartedOn' => <DateTime>,
'Statistics' => [
'FailedActions' => <integer>,
'RunningActions' => <integer>,
'StoppedActions' => <integer>,
'SucceededActions' => <integer>,
'TimeoutActions' => <integer>,
'TotalActions' => <integer>,
],
'Status' => 'RUNNING|COMPLETED',
'WorkflowRunId' => '<string>',
'WorkflowRunProperties' => ['<string>', ...],
],
'Name' => '<string>',
],
]
Result Details
Members
- Workflow
-
- Type: Workflow structure
The resource metadata for the workflow.
Errors
-
The input provided was not valid.
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
GetWorkflowRun
$result = $client->getWorkflowRun([/* ... */]); $promise = $client->getWorkflowRunAsync([/* ... */]);
Retrieves the metadata for a given workflow run.
Parameter Syntax
$result = $client->getWorkflowRun([
'IncludeGraph' => true || false,
'Name' => '<string>', // REQUIRED
'RunId' => '<string>', // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'Run' => [
'CompletedOn' => <DateTime>,
'Graph' => [
'Edges' => [
[
'DestinationId' => '<string>',
'SourceId' => '<string>',
],
// ...
],
'Nodes' => [
[
'CrawlerDetails' => [
'Crawls' => [
[
'CompletedOn' => <DateTime>,
'ErrorMessage' => '<string>',
'LogGroup' => '<string>',
'LogStream' => '<string>',
'StartedOn' => <DateTime>,
'State' => 'RUNNING|SUCCEEDED|CANCELLED|FAILED',
],
// ...
],
],
'JobDetails' => [
'JobRuns' => [
[
'AllocatedCapacity' => <integer>,
'Arguments' => ['<string>', ...],
'Attempt' => <integer>,
'CompletedOn' => <DateTime>,
'ErrorMessage' => '<string>',
'ExecutionTime' => <integer>,
'GlueVersion' => '<string>',
'Id' => '<string>',
'JobName' => '<string>',
'JobRunState' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
'LastModifiedOn' => <DateTime>,
'LogGroupName' => '<string>',
'MaxCapacity' => <float>,
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'NumberOfWorkers' => <integer>,
'PredecessorRuns' => [
[
'JobName' => '<string>',
'RunId' => '<string>',
],
// ...
],
'PreviousRunId' => '<string>',
'SecurityConfiguration' => '<string>',
'StartedOn' => <DateTime>,
'Timeout' => <integer>,
'TriggerName' => '<string>',
'WorkerType' => 'Standard|G.1X|G.2X',
],
// ...
],
],
'Name' => '<string>',
'TriggerDetails' => [
'Trigger' => [
'Actions' => [
[
'Arguments' => ['<string>', ...],
'CrawlerName' => '<string>',
'JobName' => '<string>',
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'SecurityConfiguration' => '<string>',
'Timeout' => <integer>,
],
// ...
],
'Description' => '<string>',
'Id' => '<string>',
'Name' => '<string>',
'Predicate' => [
'Conditions' => [
[
'CrawlState' => 'RUNNING|SUCCEEDED|CANCELLED|FAILED',
'CrawlerName' => '<string>',
'JobName' => '<string>',
'LogicalOperator' => 'EQUALS',
'State' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
],
// ...
],
'Logical' => 'AND|ANY',
],
'Schedule' => '<string>',
'State' => 'CREATING|CREATED|ACTIVATING|ACTIVATED|DEACTIVATING|DEACTIVATED|DELETING|UPDATING',
'Type' => 'SCHEDULED|CONDITIONAL|ON_DEMAND',
'WorkflowName' => '<string>',
],
],
'Type' => 'CRAWLER|JOB|TRIGGER',
'UniqueId' => '<string>',
],
// ...
],
],
'Name' => '<string>',
'StartedOn' => <DateTime>,
'Statistics' => [
'FailedActions' => <integer>,
'RunningActions' => <integer>,
'StoppedActions' => <integer>,
'SucceededActions' => <integer>,
'TimeoutActions' => <integer>,
'TotalActions' => <integer>,
],
'Status' => 'RUNNING|COMPLETED',
'WorkflowRunId' => '<string>',
'WorkflowRunProperties' => ['<string>', ...],
],
]
Result Details
Members
- Run
-
- Type: WorkflowRun structure
The requested workflow run metadata.
Errors
-
The input provided was not valid.
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
GetWorkflowRunProperties
$result = $client->getWorkflowRunProperties([/* ... */]); $promise = $client->getWorkflowRunPropertiesAsync([/* ... */]);
Retrieves the workflow run properties which were set during the run.
Parameter Syntax
$result = $client->getWorkflowRunProperties([
'Name' => '<string>', // REQUIRED
'RunId' => '<string>', // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'RunProperties' => ['<string>', ...],
]
Result Details
Members
Errors
-
The input provided was not valid.
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
GetWorkflowRuns
$result = $client->getWorkflowRuns([/* ... */]); $promise = $client->getWorkflowRunsAsync([/* ... */]);
Retrieves metadata for all runs of a given workflow.
Parameter Syntax
$result = $client->getWorkflowRuns([
'IncludeGraph' => true || false,
'MaxResults' => <integer>,
'Name' => '<string>', // REQUIRED
'NextToken' => '<string>',
]);
Parameter Details
Members
- IncludeGraph
-
- Type: boolean
Specifies whether to include the workflow graph in response or not.
- MaxResults
-
- Type: int
The maximum number of workflow runs to be included in the response.
- Name
-
- Required: Yes
- Type: string
Name of the workflow whose metadata of runs should be returned.
- NextToken
-
- Type: string
The maximum size of the response.
Result Syntax
[
'NextToken' => '<string>',
'Runs' => [
[
'CompletedOn' => <DateTime>,
'Graph' => [
'Edges' => [
[
'DestinationId' => '<string>',
'SourceId' => '<string>',
],
// ...
],
'Nodes' => [
[
'CrawlerDetails' => [
'Crawls' => [
[
'CompletedOn' => <DateTime>,
'ErrorMessage' => '<string>',
'LogGroup' => '<string>',
'LogStream' => '<string>',
'StartedOn' => <DateTime>,
'State' => 'RUNNING|SUCCEEDED|CANCELLED|FAILED',
],
// ...
],
],
'JobDetails' => [
'JobRuns' => [
[
'AllocatedCapacity' => <integer>,
'Arguments' => ['<string>', ...],
'Attempt' => <integer>,
'CompletedOn' => <DateTime>,
'ErrorMessage' => '<string>',
'ExecutionTime' => <integer>,
'GlueVersion' => '<string>',
'Id' => '<string>',
'JobName' => '<string>',
'JobRunState' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
'LastModifiedOn' => <DateTime>,
'LogGroupName' => '<string>',
'MaxCapacity' => <float>,
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'NumberOfWorkers' => <integer>,
'PredecessorRuns' => [
[
'JobName' => '<string>',
'RunId' => '<string>',
],
// ...
],
'PreviousRunId' => '<string>',
'SecurityConfiguration' => '<string>',
'StartedOn' => <DateTime>,
'Timeout' => <integer>,
'TriggerName' => '<string>',
'WorkerType' => 'Standard|G.1X|G.2X',
],
// ...
],
],
'Name' => '<string>',
'TriggerDetails' => [
'Trigger' => [
'Actions' => [
[
'Arguments' => ['<string>', ...],
'CrawlerName' => '<string>',
'JobName' => '<string>',
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'SecurityConfiguration' => '<string>',
'Timeout' => <integer>,
],
// ...
],
'Description' => '<string>',
'Id' => '<string>',
'Name' => '<string>',
'Predicate' => [
'Conditions' => [
[
'CrawlState' => 'RUNNING|SUCCEEDED|CANCELLED|FAILED',
'CrawlerName' => '<string>',
'JobName' => '<string>',
'LogicalOperator' => 'EQUALS',
'State' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
],
// ...
],
'Logical' => 'AND|ANY',
],
'Schedule' => '<string>',
'State' => 'CREATING|CREATED|ACTIVATING|ACTIVATED|DEACTIVATING|DEACTIVATED|DELETING|UPDATING',
'Type' => 'SCHEDULED|CONDITIONAL|ON_DEMAND',
'WorkflowName' => '<string>',
],
],
'Type' => 'CRAWLER|JOB|TRIGGER',
'UniqueId' => '<string>',
],
// ...
],
],
'Name' => '<string>',
'StartedOn' => <DateTime>,
'Statistics' => [
'FailedActions' => <integer>,
'RunningActions' => <integer>,
'StoppedActions' => <integer>,
'SucceededActions' => <integer>,
'TimeoutActions' => <integer>,
'TotalActions' => <integer>,
],
'Status' => 'RUNNING|COMPLETED',
'WorkflowRunId' => '<string>',
'WorkflowRunProperties' => ['<string>', ...],
],
// ...
],
]
Result Details
Members
- NextToken
-
- Type: string
A continuation token, if not all requested workflow runs have been returned.
- Runs
-
- Type: Array of WorkflowRun structures
A list of workflow run metadata objects.
Errors
-
The input provided was not valid.
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
ImportCatalogToGlue
$result = $client->importCatalogToGlue([/* ... */]); $promise = $client->importCatalogToGlueAsync([/* ... */]);
Imports an existing Amazon Athena Data Catalog to AWS Glue
Parameter Syntax
$result = $client->importCatalogToGlue([
'CatalogId' => '<string>',
]);
Parameter Details
Members
Result Syntax
[]
Result Details
Errors
-
An internal service error occurred.
-
The operation timed out.
ListCrawlers
$result = $client->listCrawlers([/* ... */]); $promise = $client->listCrawlersAsync([/* ... */]);
Retrieves the names of all crawler resources in this AWS account, or the resources with the specified tag. This operation allows you to see which resources are available in your account, and their names.
This operation takes the optional Tags field, which you can use as a filter on the response so that tagged resources can be retrieved as a group. If you choose to use tags filtering, only resources with the tag are retrieved.
Parameter Syntax
$result = $client->listCrawlers([
'MaxResults' => <integer>,
'NextToken' => '<string>',
'Tags' => ['<string>', ...],
]);
Parameter Details
Members
Result Syntax
[
'CrawlerNames' => ['<string>', ...],
'NextToken' => '<string>',
]
Result Details
Members
Errors
-
The operation timed out.
ListDevEndpoints
$result = $client->listDevEndpoints([/* ... */]); $promise = $client->listDevEndpointsAsync([/* ... */]);
Retrieves the names of all DevEndpoint resources in this AWS account, or the resources with the specified tag. This operation allows you to see which resources are available in your account, and their names.
This operation takes the optional Tags field, which you can use as a filter on the response so that tagged resources can be retrieved as a group. If you choose to use tags filtering, only resources with the tag are retrieved.
Parameter Syntax
$result = $client->listDevEndpoints([
'MaxResults' => <integer>,
'NextToken' => '<string>',
'Tags' => ['<string>', ...],
]);
Parameter Details
Members
Result Syntax
[
'DevEndpointNames' => ['<string>', ...],
'NextToken' => '<string>',
]
Result Details
Members
Errors
-
The input provided was not valid.
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
ListJobs
$result = $client->listJobs([/* ... */]); $promise = $client->listJobsAsync([/* ... */]);
Retrieves the names of all job resources in this AWS account, or the resources with the specified tag. This operation allows you to see which resources are available in your account, and their names.
This operation takes the optional Tags field, which you can use as a filter on the response so that tagged resources can be retrieved as a group. If you choose to use tags filtering, only resources with the tag are retrieved.
Parameter Syntax
$result = $client->listJobs([
'MaxResults' => <integer>,
'NextToken' => '<string>',
'Tags' => ['<string>', ...],
]);
Parameter Details
Members
Result Syntax
[
'JobNames' => ['<string>', ...],
'NextToken' => '<string>',
]
Result Details
Members
Errors
-
The input provided was not valid.
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
ListTriggers
$result = $client->listTriggers([/* ... */]); $promise = $client->listTriggersAsync([/* ... */]);
Retrieves the names of all trigger resources in this AWS account, or the resources with the specified tag. This operation allows you to see which resources are available in your account, and their names.
This operation takes the optional Tags field, which you can use as a filter on the response so that tagged resources can be retrieved as a group. If you choose to use tags filtering, only resources with the tag are retrieved.
Parameter Syntax
$result = $client->listTriggers([
'DependentJobName' => '<string>',
'MaxResults' => <integer>,
'NextToken' => '<string>',
'Tags' => ['<string>', ...],
]);
Parameter Details
Members
- DependentJobName
-
- Type: string
The name of the job for which to retrieve triggers. The trigger that can start this job is returned. If there is no such trigger, all triggers are returned.
- MaxResults
-
- Type: int
The maximum size of a list to return.
- NextToken
-
- Type: string
A continuation token, if this is a continuation request.
- Tags
-
- Type: Associative array of custom strings keys (TagKey) to strings
Specifies to return only these tagged resources.
Result Syntax
[
'NextToken' => '<string>',
'TriggerNames' => ['<string>', ...],
]
Result Details
Members
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
ListWorkflows
$result = $client->listWorkflows([/* ... */]); $promise = $client->listWorkflowsAsync([/* ... */]);
Lists names of workflows created in the account.
Parameter Syntax
$result = $client->listWorkflows([
'MaxResults' => <integer>,
'NextToken' => '<string>',
]);
Parameter Details
Members
Result Syntax
[
'NextToken' => '<string>',
'Workflows' => ['<string>', ...],
]
Result Details
Members
Errors
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
PutDataCatalogEncryptionSettings
$result = $client->putDataCatalogEncryptionSettings([/* ... */]); $promise = $client->putDataCatalogEncryptionSettingsAsync([/* ... */]);
Sets the security configuration for a specified catalog. After the configuration has been set, the specified encryption is applied to every catalog write thereafter.
Parameter Syntax
$result = $client->putDataCatalogEncryptionSettings([
'CatalogId' => '<string>',
'DataCatalogEncryptionSettings' => [ // REQUIRED
'ConnectionPasswordEncryption' => [
'AwsKmsKeyId' => '<string>',
'ReturnConnectionPasswordEncrypted' => true || false, // REQUIRED
],
'EncryptionAtRest' => [
'CatalogEncryptionMode' => 'DISABLED|SSE-KMS', // REQUIRED
'SseAwsKmsKeyId' => '<string>',
],
],
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog to set the security configuration for. If none is provided, the AWS account ID is used by default.
- DataCatalogEncryptionSettings
-
- Required: Yes
- Type: DataCatalogEncryptionSettings structure
The security configuration to set.
Result Syntax
[]
Result Details
Errors
-
An internal service error occurred.
-
The input provided was not valid.
-
The operation timed out.
PutResourcePolicy
$result = $client->putResourcePolicy([/* ... */]); $promise = $client->putResourcePolicyAsync([/* ... */]);
Sets the Data Catalog resource policy for access control.
Parameter Syntax
$result = $client->putResourcePolicy([
'PolicyExistsCondition' => 'MUST_EXIST|NOT_EXIST|NONE',
'PolicyHashCondition' => '<string>',
'PolicyInJson' => '<string>', // REQUIRED
]);
Parameter Details
Members
- PolicyExistsCondition
-
- Type: string
A value of
MUST_EXISTis used to update a policy. A value ofNOT_EXISTis used to create a new policy. If a value ofNONEor a null value is used, the call will not depend on the existence of a policy. - PolicyHashCondition
-
- Type: string
The hash value returned when the previous policy was set using
PutResourcePolicy. Its purpose is to prevent concurrent modifications of a policy. Do not use this parameter if no previous policy has been set. - PolicyInJson
-
- Required: Yes
- Type: string
Contains the policy document to set, in JSON format.
Result Syntax
[
'PolicyHash' => '<string>',
]
Result Details
Members
Errors
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
-
The input provided was not valid.
-
ConditionCheckFailureException:
A specified condition was not satisfied.
PutWorkflowRunProperties
$result = $client->putWorkflowRunProperties([/* ... */]); $promise = $client->putWorkflowRunPropertiesAsync([/* ... */]);
Puts the specified workflow run properties for the given workflow run. If a property already exists for the specified run, then it overrides the value otherwise adds the property to existing properties.
Parameter Syntax
$result = $client->putWorkflowRunProperties([
'Name' => '<string>', // REQUIRED
'RunId' => '<string>', // REQUIRED
'RunProperties' => ['<string>', ...], // REQUIRED
]);
Parameter Details
Members
- Name
-
- Required: Yes
- Type: string
Name of the workflow which was run.
- RunId
-
- Required: Yes
- Type: string
The ID of the workflow run for which the run properties should be updated.
- RunProperties
-
- Required: Yes
- Type: Associative array of custom strings keys (IdString) to strings
The properties to put for the specified run.
Result Syntax
[]
Result Details
Errors
-
A resource to be created or added already exists.
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
-
ResourceNumberLimitExceededException:
A resource numerical limit was exceeded.
-
ConcurrentModificationException:
Two processes are trying to modify a resource simultaneously.
ResetJobBookmark
$result = $client->resetJobBookmark([/* ... */]); $promise = $client->resetJobBookmarkAsync([/* ... */]);
Resets a bookmark entry.
Parameter Syntax
$result = $client->resetJobBookmark([
'JobName' => '<string>', // REQUIRED
'RunId' => '<string>',
]);
Parameter Details
Members
Result Syntax
[
'JobBookmarkEntry' => [
'Attempt' => <integer>,
'JobBookmark' => '<string>',
'JobName' => '<string>',
'PreviousRunId' => '<string>',
'Run' => <integer>,
'RunId' => '<string>',
'Version' => <integer>,
],
]
Result Details
Members
- JobBookmarkEntry
-
- Type: JobBookmarkEntry structure
The reset bookmark entry.
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
SearchTables
$result = $client->searchTables([/* ... */]); $promise = $client->searchTablesAsync([/* ... */]);
Searches a set of tables based on properties in the table metadata as well as on the parent database. You can search against text or filter conditions.
You can only get tables that you have access to based on the security policies defined in Lake Formation. You need at least a read-only access to the table for it to be returned. If you do not have access to all the columns in the table, these columns will not be searched against when returning the list of tables back to you. If you have access to the columns but not the data in the columns, those columns and the associated metadata for those columns will be included in the search.
Parameter Syntax
$result = $client->searchTables([
'CatalogId' => '<string>',
'Filters' => [
[
'Comparator' => 'EQUALS|GREATER_THAN|LESS_THAN|GREATER_THAN_EQUALS|LESS_THAN_EQUALS',
'Key' => '<string>',
'Value' => '<string>',
],
// ...
],
'MaxResults' => <integer>,
'NextToken' => '<string>',
'SearchText' => '<string>',
'SortCriteria' => [
[
'FieldName' => '<string>',
'Sort' => 'ASC|DESC',
],
// ...
],
]);
Parameter Details
Members
- CatalogId
-
- Type: string
A unique identifier, consisting of
account_id/datalake. - Filters
-
- Type: Array of PropertyPredicate structures
A list of key-value pairs, and a comparator used to filter the search results. Returns all entities matching the predicate.
- MaxResults
-
- Type: int
The maximum number of tables to return in a single response.
- NextToken
-
- Type: string
A continuation token, included if this is a continuation call.
- SearchText
-
- Type: string
A string used for a text search.
Specifying a value in quotes filters based on an exact match to the value.
- SortCriteria
-
- Type: Array of SortCriterion structures
A list of criteria for sorting the results by a field name, in an ascending or descending order.
Result Syntax
[
'NextToken' => '<string>',
'TableList' => [
[
'CreateTime' => <DateTime>,
'CreatedBy' => '<string>',
'DatabaseName' => '<string>',
'Description' => '<string>',
'IsRegisteredWithLakeFormation' => true || false,
'LastAccessTime' => <DateTime>,
'LastAnalyzedTime' => <DateTime>,
'Name' => '<string>',
'Owner' => '<string>',
'Parameters' => ['<string>', ...],
'PartitionKeys' => [
[
'Comment' => '<string>',
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'Type' => '<string>',
],
// ...
],
'Retention' => <integer>,
'StorageDescriptor' => [
'BucketColumns' => ['<string>', ...],
'Columns' => [
[
'Comment' => '<string>',
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'Type' => '<string>',
],
// ...
],
'Compressed' => true || false,
'InputFormat' => '<string>',
'Location' => '<string>',
'NumberOfBuckets' => <integer>,
'OutputFormat' => '<string>',
'Parameters' => ['<string>', ...],
'SerdeInfo' => [
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'SerializationLibrary' => '<string>',
],
'SkewedInfo' => [
'SkewedColumnNames' => ['<string>', ...],
'SkewedColumnValueLocationMaps' => ['<string>', ...],
'SkewedColumnValues' => ['<string>', ...],
],
'SortColumns' => [
[
'Column' => '<string>',
'SortOrder' => <integer>,
],
// ...
],
'StoredAsSubDirectories' => true || false,
],
'TableType' => '<string>',
'UpdateTime' => <DateTime>,
'ViewExpandedText' => '<string>',
'ViewOriginalText' => '<string>',
],
// ...
],
]
Result Details
Members
- NextToken
-
- Type: string
A continuation token, present if the current list segment is not the last.
- TableList
-
- Type: Array of Table structures
A list of the requested
Tableobjects. TheSearchTablesresponse returns only the tables that you have access to.
Errors
-
An internal service error occurred.
-
The input provided was not valid.
-
The operation timed out.
StartCrawler
$result = $client->startCrawler([/* ... */]); $promise = $client->startCrawlerAsync([/* ... */]);
Starts a crawl using the specified crawler, regardless of what is scheduled. If the crawler is already running, returns a CrawlerRunningException.
Parameter Syntax
$result = $client->startCrawler([
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
The operation cannot be performed because the crawler is already running.
-
The operation timed out.
StartCrawlerSchedule
$result = $client->startCrawlerSchedule([/* ... */]); $promise = $client->startCrawlerScheduleAsync([/* ... */]);
Changes the schedule state of the specified crawler to SCHEDULED, unless the crawler is already running or the schedule state is already SCHEDULED.
Parameter Syntax
$result = $client->startCrawlerSchedule([
'CrawlerName' => '<string>', // REQUIRED
]);
Parameter Details
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
The specified scheduler is already running.
-
SchedulerTransitioningException:
The specified scheduler is transitioning.
-
There is no applicable schedule.
-
The operation timed out.
StartExportLabelsTaskRun
$result = $client->startExportLabelsTaskRun([/* ... */]); $promise = $client->startExportLabelsTaskRunAsync([/* ... */]);
Begins an asynchronous task to export all labeled data for a particular transform. This task is the only label-related API call that is not part of the typical active learning workflow. You typically use StartExportLabelsTaskRun when you want to work with all of your existing labels at the same time, such as when you want to remove or change labels that were previously submitted as truth. This API operation accepts the TransformId whose labels you want to export and an Amazon Simple Storage Service (Amazon S3) path to export the labels to. The operation returns a TaskRunId. You can check on the status of your task run by calling the GetMLTaskRun API.
Parameter Syntax
$result = $client->startExportLabelsTaskRun([
'OutputS3Path' => '<string>', // REQUIRED
'TransformId' => '<string>', // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'TaskRunId' => '<string>',
]
Result Details
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
The operation timed out.
-
An internal service error occurred.
StartImportLabelsTaskRun
$result = $client->startImportLabelsTaskRun([/* ... */]); $promise = $client->startImportLabelsTaskRunAsync([/* ... */]);
Enables you to provide additional labels (examples of truth) to be used to teach the machine learning transform and improve its quality. This API operation is generally used as part of the active learning workflow that starts with the StartMLLabelingSetGenerationTaskRun call and that ultimately results in improving the quality of your machine learning transform.
After the StartMLLabelingSetGenerationTaskRun finishes, AWS Glue machine learning will have generated a series of questions for humans to answer. (Answering these questions is often called 'labeling' in the machine learning workflows). In the case of the FindMatches transform, these questions are of the form, “What is the correct way to group these rows together into groups composed entirely of matching records?” After the labeling process is finished, users upload their answers/labels with a call to StartImportLabelsTaskRun. After StartImportLabelsTaskRun finishes, all future runs of the machine learning transform use the new and improved labels and perform a higher-quality transformation.
By default, StartMLLabelingSetGenerationTaskRun continually learns from and combines all labels that you upload unless you set Replace to true. If you set Replace to true, StartImportLabelsTaskRun deletes and forgets all previously uploaded labels and learns only from the exact set that you upload. Replacing labels can be helpful if you realize that you previously uploaded incorrect labels, and you believe that they are having a negative effect on your transform quality.
You can check on the status of your task run by calling the GetMLTaskRun operation.
Parameter Syntax
$result = $client->startImportLabelsTaskRun([
'InputS3Path' => '<string>', // REQUIRED
'ReplaceAllLabels' => true || false,
'TransformId' => '<string>', // REQUIRED
]);
Parameter Details
Members
- InputS3Path
-
- Required: Yes
- Type: string
The Amazon Simple Storage Service (Amazon S3) path from where you import the labels.
- ReplaceAllLabels
-
- Type: boolean
Indicates whether to overwrite your existing labels.
- TransformId
-
- Required: Yes
- Type: string
The unique identifier of the machine learning transform.
Result Syntax
[
'TaskRunId' => '<string>',
]
Result Details
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
The operation timed out.
-
ResourceNumberLimitExceededException:
A resource numerical limit was exceeded.
-
An internal service error occurred.
StartJobRun
$result = $client->startJobRun([/* ... */]); $promise = $client->startJobRunAsync([/* ... */]);
Starts a job run using a job definition.
Parameter Syntax
$result = $client->startJobRun([
'AllocatedCapacity' => <integer>,
'Arguments' => ['<string>', ...],
'JobName' => '<string>', // REQUIRED
'JobRunId' => '<string>',
'MaxCapacity' => <float>,
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'NumberOfWorkers' => <integer>,
'SecurityConfiguration' => '<string>',
'Timeout' => <integer>,
'WorkerType' => 'Standard|G.1X|G.2X',
]);
Parameter Details
Members
- AllocatedCapacity
-
- Type: int
This field is deprecated. Use
MaxCapacityinstead.The number of AWS Glue data processing units (DPUs) to allocate to this JobRun. From 2 to 100 DPUs can be allocated; the default is 10. A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB of memory. For more information, see the AWS Glue pricing page.
- Arguments
-
- Type: Associative array of custom strings keys (GenericString) to strings
The job arguments specifically for this run. For this job run, they replace the default arguments set in the job definition itself.
You can specify arguments here that your own job-execution script consumes, as well as arguments that AWS Glue itself consumes.
For information about how to specify and consume your own Job arguments, see the Calling AWS Glue APIs in Python topic in the developer guide.
For information about the key-value pairs that AWS Glue consumes to set up your job, see the Special Parameters Used by AWS Glue topic in the developer guide.
- JobName
-
- Required: Yes
- Type: string
The name of the job definition to use.
- JobRunId
-
- Type: string
The ID of a previous
JobRunto retry. - MaxCapacity
-
- Type: double
The number of AWS Glue data processing units (DPUs) that can be allocated when this job runs. A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB of memory. For more information, see the AWS Glue pricing page.
Do not set
Max Capacityif usingWorkerTypeandNumberOfWorkers.The value that can be allocated for
MaxCapacitydepends on whether you are running a Python shell job, or an Apache Spark ETL job:-
When you specify a Python shell job (
JobCommand.Name="pythonshell"), you can allocate either 0.0625 or 1 DPU. The default is 0.0625 DPU. -
When you specify an Apache Spark ETL job (
JobCommand.Name="glueetl"), you can allocate from 2 to 100 DPUs. The default is 10 DPUs. This job type cannot have a fractional DPU allocation.
- NotificationProperty
-
- Type: NotificationProperty structure
Specifies configuration properties of a job run notification.
- NumberOfWorkers
-
- Type: int
The number of workers of a defined
workerTypethat are allocated when a job runs.The maximum number of workers you can define are 299 for
G.1X, and 149 forG.2X. - SecurityConfiguration
-
- Type: string
The name of the
SecurityConfigurationstructure to be used with this job run. - Timeout
-
- Type: int
The
JobRuntimeout in minutes. This is the maximum time that a job run can consume resources before it is terminated and entersTIMEOUTstatus. The default is 2,880 minutes (48 hours). This overrides the timeout value set in the parent job. - WorkerType
-
- Type: string
The type of predefined worker that is allocated when a job runs. Accepts a value of Standard, G.1X, or G.2X.
-
For the
Standardworker type, each worker provides 4 vCPU, 16 GB of memory and a 50GB disk, and 2 executors per worker. -
For the
G.1Xworker type, each worker provides 4 vCPU, 16 GB of memory and a 64GB disk, and 1 executor per worker. -
For the
G.2Xworker type, each worker provides 8 vCPU, 32 GB of memory and a 128GB disk, and 1 executor per worker.
Result Syntax
[
'JobRunId' => '<string>',
]
Result Details
Errors
-
The input provided was not valid.
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
-
ResourceNumberLimitExceededException:
A resource numerical limit was exceeded.
-
ConcurrentRunsExceededException:
Too many jobs are being run concurrently.
StartMLEvaluationTaskRun
$result = $client->startMLEvaluationTaskRun([/* ... */]); $promise = $client->startMLEvaluationTaskRunAsync([/* ... */]);
Starts a task to estimate the quality of the transform.
When you provide label sets as examples of truth, AWS Glue machine learning uses some of those examples to learn from them. The rest of the labels are used as a test to estimate quality.
Returns a unique identifier for the run. You can call GetMLTaskRun to get more information about the stats of the EvaluationTaskRun.
Parameter Syntax
$result = $client->startMLEvaluationTaskRun([
'TransformId' => '<string>', // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'TaskRunId' => '<string>',
]
Result Details
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
The operation timed out.
-
An internal service error occurred.
-
ConcurrentRunsExceededException:
Too many jobs are being run concurrently.
-
The machine learning transform is not ready to run.
StartMLLabelingSetGenerationTaskRun
$result = $client->startMLLabelingSetGenerationTaskRun([/* ... */]); $promise = $client->startMLLabelingSetGenerationTaskRunAsync([/* ... */]);
Starts the active learning workflow for your machine learning transform to improve the transform's quality by generating label sets and adding labels.
When the StartMLLabelingSetGenerationTaskRun finishes, AWS Glue will have generated a "labeling set" or a set of questions for humans to answer.
In the case of the FindMatches transform, these questions are of the form, “What is the correct way to group these rows together into groups composed entirely of matching records?”
After the labeling process is finished, you can upload your labels with a call to StartImportLabelsTaskRun. After StartImportLabelsTaskRun finishes, all future runs of the machine learning transform will use the new and improved labels and perform a higher-quality transformation.
Parameter Syntax
$result = $client->startMLLabelingSetGenerationTaskRun([
'OutputS3Path' => '<string>', // REQUIRED
'TransformId' => '<string>', // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'TaskRunId' => '<string>',
]
Result Details
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
The operation timed out.
-
An internal service error occurred.
-
ConcurrentRunsExceededException:
Too many jobs are being run concurrently.
StartTrigger
$result = $client->startTrigger([/* ... */]); $promise = $client->startTriggerAsync([/* ... */]);
Starts an existing trigger. See Triggering Jobs for information about how different types of trigger are started.
Parameter Syntax
$result = $client->startTrigger([
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Result Syntax
[
'Name' => '<string>',
]
Result Details
Errors
-
The input provided was not valid.
-
An internal service error occurred.
-
A specified entity does not exist
-
The operation timed out.
-
ResourceNumberLimitExceededException:
A resource numerical limit was exceeded.
-
ConcurrentRunsExceededException:
Too many jobs are being run concurrently.
StartWorkflowRun
$result = $client->startWorkflowRun([/* ... */]); $promise = $client->startWorkflowRunAsync([/* ... */]);
Starts a new run of the specified workflow.
Parameter Syntax
$result = $client->startWorkflowRun([
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Result Syntax
[
'RunId' => '<string>',
]
Result Details
Errors
-
The input provided was not valid.
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
-
ResourceNumberLimitExceededException:
A resource numerical limit was exceeded.
-
ConcurrentRunsExceededException:
Too many jobs are being run concurrently.
StopCrawler
$result = $client->stopCrawler([/* ... */]); $promise = $client->stopCrawlerAsync([/* ... */]);
If the specified crawler is running, stops the crawl.
Parameter Syntax
$result = $client->stopCrawler([
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
The specified crawler is not running.
-
The specified crawler is stopping.
-
The operation timed out.
StopCrawlerSchedule
$result = $client->stopCrawlerSchedule([/* ... */]); $promise = $client->stopCrawlerScheduleAsync([/* ... */]);
Sets the schedule state of the specified crawler to NOT_SCHEDULED, but does not stop the crawler if it is already running.
Parameter Syntax
$result = $client->stopCrawlerSchedule([
'CrawlerName' => '<string>', // REQUIRED
]);
Parameter Details
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
The specified scheduler is not running.
-
SchedulerTransitioningException:
The specified scheduler is transitioning.
-
The operation timed out.
StopTrigger
$result = $client->stopTrigger([/* ... */]); $promise = $client->stopTriggerAsync([/* ... */]);
Stops a specified trigger.
Parameter Syntax
$result = $client->stopTrigger([
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Result Syntax
[
'Name' => '<string>',
]
Result Details
Errors
-
The input provided was not valid.
-
An internal service error occurred.
-
A specified entity does not exist
-
The operation timed out.
-
ConcurrentModificationException:
Two processes are trying to modify a resource simultaneously.
TagResource
$result = $client->tagResource([/* ... */]); $promise = $client->tagResourceAsync([/* ... */]);
Adds tags to a resource. A tag is a label you can assign to an AWS resource. In AWS Glue, you can tag only certain resources. For information about what resources you can tag, see AWS Tags in AWS Glue.
Parameter Syntax
$result = $client->tagResource([
'ResourceArn' => '<string>', // REQUIRED
'TagsToAdd' => ['<string>', ...], // REQUIRED
]);
Parameter Details
Members
- ResourceArn
-
- Required: Yes
- Type: string
The ARN of the AWS Glue resource to which to add the tags. For more information about AWS Glue resource ARNs, see the AWS Glue ARN string pattern.
- TagsToAdd
-
- Required: Yes
- Type: Associative array of custom strings keys (TagKey) to strings
Tags to add to this resource.
Result Syntax
[]
Result Details
Errors
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
-
A specified entity does not exist
UntagResource
$result = $client->untagResource([/* ... */]); $promise = $client->untagResourceAsync([/* ... */]);
Removes tags from a resource.
Parameter Syntax
$result = $client->untagResource([
'ResourceArn' => '<string>', // REQUIRED
'TagsToRemove' => ['<string>', ...], // REQUIRED
]);
Parameter Details
Members
Result Syntax
[]
Result Details
Errors
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
-
A specified entity does not exist
UpdateClassifier
$result = $client->updateClassifier([/* ... */]); $promise = $client->updateClassifierAsync([/* ... */]);
Modifies an existing classifier (a GrokClassifier, an XMLClassifier, a JsonClassifier, or a CsvClassifier, depending on which field is present).
Parameter Syntax
$result = $client->updateClassifier([
'CsvClassifier' => [
'AllowSingleColumn' => true || false,
'ContainsHeader' => 'UNKNOWN|PRESENT|ABSENT',
'Delimiter' => '<string>',
'DisableValueTrimming' => true || false,
'Header' => ['<string>', ...],
'Name' => '<string>', // REQUIRED
'QuoteSymbol' => '<string>',
],
'GrokClassifier' => [
'Classification' => '<string>',
'CustomPatterns' => '<string>',
'GrokPattern' => '<string>',
'Name' => '<string>', // REQUIRED
],
'JsonClassifier' => [
'JsonPath' => '<string>',
'Name' => '<string>', // REQUIRED
],
'XMLClassifier' => [
'Classification' => '<string>',
'Name' => '<string>', // REQUIRED
'RowTag' => '<string>',
],
]);
Parameter Details
Members
- CsvClassifier
-
- Type: UpdateCsvClassifierRequest structure
A
CsvClassifierobject with updated fields. - GrokClassifier
-
- Type: UpdateGrokClassifierRequest structure
A
GrokClassifierobject with updated fields. - JsonClassifier
-
- Type: UpdateJsonClassifierRequest structure
A
JsonClassifierobject with updated fields. - XMLClassifier
-
- Type: UpdateXMLClassifierRequest structure
An
XMLClassifierobject with updated fields.
Result Syntax
[]
Result Details
Errors
-
The input provided was not valid.
-
There was a version conflict.
-
A specified entity does not exist
-
The operation timed out.
UpdateConnection
$result = $client->updateConnection([/* ... */]); $promise = $client->updateConnectionAsync([/* ... */]);
Updates a connection definition in the Data Catalog.
Parameter Syntax
$result = $client->updateConnection([
'CatalogId' => '<string>',
'ConnectionInput' => [ // REQUIRED
'ConnectionProperties' => ['<string>', ...], // REQUIRED
'ConnectionType' => 'JDBC|SFTP', // REQUIRED
'Description' => '<string>',
'MatchCriteria' => ['<string>', ...],
'Name' => '<string>', // REQUIRED
'PhysicalConnectionRequirements' => [
'AvailabilityZone' => '<string>',
'SecurityGroupIdList' => ['<string>', ...],
'SubnetId' => '<string>',
],
],
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog in which the connection resides. If none is provided, the AWS account ID is used by default.
- ConnectionInput
-
- Required: Yes
- Type: ConnectionInput structure
A
ConnectionInputobject that redefines the connection in question. - Name
-
- Required: Yes
- Type: string
The name of the connection definition to update.
Result Syntax
[]
Result Details
Errors
-
The input provided was not valid.
-
A specified entity does not exist
-
The operation timed out.
-
The input provided was not valid.
-
An encryption operation failed.
UpdateCrawler
$result = $client->updateCrawler([/* ... */]); $promise = $client->updateCrawlerAsync([/* ... */]);
Updates a crawler. If a crawler is running, you must stop it using StopCrawler before updating it.
Parameter Syntax
$result = $client->updateCrawler([
'Classifiers' => ['<string>', ...],
'Configuration' => '<string>',
'CrawlerSecurityConfiguration' => '<string>',
'DatabaseName' => '<string>',
'Description' => '<string>',
'Name' => '<string>', // REQUIRED
'Role' => '<string>',
'Schedule' => '<string>',
'SchemaChangePolicy' => [
'DeleteBehavior' => 'LOG|DELETE_FROM_DATABASE|DEPRECATE_IN_DATABASE',
'UpdateBehavior' => 'LOG|UPDATE_IN_DATABASE',
],
'TablePrefix' => '<string>',
'Targets' => [
'CatalogTargets' => [
[
'DatabaseName' => '<string>', // REQUIRED
'Tables' => ['<string>', ...], // REQUIRED
],
// ...
],
'DynamoDBTargets' => [
[
'Path' => '<string>',
],
// ...
],
'JdbcTargets' => [
[
'ConnectionName' => '<string>',
'Exclusions' => ['<string>', ...],
'Path' => '<string>',
],
// ...
],
'S3Targets' => [
[
'Exclusions' => ['<string>', ...],
'Path' => '<string>',
],
// ...
],
],
]);
Parameter Details
Members
- Classifiers
-
- Type: Array of strings
A list of custom classifiers that the user has registered. By default, all built-in classifiers are included in a crawl, but these custom classifiers always override the default classifiers for a given classification.
- Configuration
-
- Type: string
The crawler configuration information. This versioned JSON string allows users to specify aspects of a crawler's behavior. For more information, see Configuring a Crawler.
- CrawlerSecurityConfiguration
-
- Type: string
The name of the
SecurityConfigurationstructure to be used by this crawler. - DatabaseName
-
- Type: string
The AWS Glue database where results are stored, such as:
arn:aws:daylight:us-east-1::database/sometable/*. - Description
-
- Type: string
A description of the new crawler.
- Name
-
- Required: Yes
- Type: string
Name of the new crawler.
- Role
-
- Type: string
The IAM role or Amazon Resource Name (ARN) of an IAM role that is used by the new crawler to access customer resources.
- Schedule
-
- Type: string
A
cronexpression used to specify the schedule. For more information, see Time-Based Schedules for Jobs and Crawlers. For example, to run something every day at 12:15 UTC, specifycron(15 12 * * ? *). - SchemaChangePolicy
-
- Type: SchemaChangePolicy structure
The policy for the crawler's update and deletion behavior.
- TablePrefix
-
- Type: string
The table prefix used for catalog tables that are created.
- Targets
-
- Type: CrawlerTargets structure
A list of targets to crawl.
Result Syntax
[]
Result Details
Errors
-
The input provided was not valid.
-
There was a version conflict.
-
A specified entity does not exist
-
The operation cannot be performed because the crawler is already running.
-
The operation timed out.
UpdateCrawlerSchedule
$result = $client->updateCrawlerSchedule([/* ... */]); $promise = $client->updateCrawlerScheduleAsync([/* ... */]);
Updates the schedule of a crawler using a cron expression.
Parameter Syntax
$result = $client->updateCrawlerSchedule([
'CrawlerName' => '<string>', // REQUIRED
'Schedule' => '<string>',
]);
Parameter Details
Members
- CrawlerName
-
- Required: Yes
- Type: string
The name of the crawler whose schedule to update.
- Schedule
-
- Type: string
The updated
cronexpression used to specify the schedule. For more information, see Time-Based Schedules for Jobs and Crawlers. For example, to run something every day at 12:15 UTC, specifycron(15 12 * * ? *).
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
There was a version conflict.
-
SchedulerTransitioningException:
The specified scheduler is transitioning.
-
The operation timed out.
UpdateDatabase
$result = $client->updateDatabase([/* ... */]); $promise = $client->updateDatabaseAsync([/* ... */]);
Updates an existing database definition in a Data Catalog.
Parameter Syntax
$result = $client->updateDatabase([
'CatalogId' => '<string>',
'DatabaseInput' => [ // REQUIRED
'CreateTableDefaultPermissions' => [
[
'Permissions' => ['<string>', ...],
'Principal' => [
'DataLakePrincipalIdentifier' => '<string>',
],
],
// ...
],
'Description' => '<string>',
'LocationUri' => '<string>',
'Name' => '<string>', // REQUIRED
'Parameters' => ['<string>', ...],
],
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog in which the metadata database resides. If none is provided, the AWS account ID is used by default.
- DatabaseInput
-
- Required: Yes
- Type: DatabaseInput structure
A
DatabaseInputobject specifying the new definition of the metadata database in the catalog. - Name
-
- Required: Yes
- Type: string
The name of the database to update in the catalog. For Hive compatibility, this is folded to lowercase.
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
-
An encryption operation failed.
UpdateDevEndpoint
$result = $client->updateDevEndpoint([/* ... */]); $promise = $client->updateDevEndpointAsync([/* ... */]);
Updates a specified development endpoint.
Parameter Syntax
$result = $client->updateDevEndpoint([
'AddArguments' => ['<string>', ...],
'AddPublicKeys' => ['<string>', ...],
'CustomLibraries' => [
'ExtraJarsS3Path' => '<string>',
'ExtraPythonLibsS3Path' => '<string>',
],
'DeleteArguments' => ['<string>', ...],
'DeletePublicKeys' => ['<string>', ...],
'EndpointName' => '<string>', // REQUIRED
'PublicKey' => '<string>',
'UpdateEtlLibraries' => true || false,
]);
Parameter Details
Members
- AddArguments
-
- Type: Associative array of custom strings keys (GenericString) to strings
The map of arguments to add the map of arguments used to configure the
DevEndpoint. - AddPublicKeys
-
- Type: Array of strings
The list of public keys for the
DevEndpointto use. - CustomLibraries
-
- Type: DevEndpointCustomLibraries structure
Custom Python or Java libraries to be loaded in the
DevEndpoint. - DeleteArguments
-
- Type: Array of strings
The list of argument keys to be deleted from the map of arguments used to configure the
DevEndpoint. - DeletePublicKeys
-
- Type: Array of strings
The list of public keys to be deleted from the
DevEndpoint. - EndpointName
-
- Required: Yes
- Type: string
The name of the
DevEndpointto be updated. - PublicKey
-
- Type: string
The public key for the
DevEndpointto use. - UpdateEtlLibraries
-
- Type: boolean
Trueif the list of custom libraries to be loaded in the development endpoint needs to be updated, orFalseif otherwise.
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
-
The input provided was not valid.
-
A value could not be validated.
UpdateJob
$result = $client->updateJob([/* ... */]); $promise = $client->updateJobAsync([/* ... */]);
Updates an existing job definition.
Parameter Syntax
$result = $client->updateJob([
'JobName' => '<string>', // REQUIRED
'JobUpdate' => [ // REQUIRED
'AllocatedCapacity' => <integer>,
'Command' => [
'Name' => '<string>',
'PythonVersion' => '<string>',
'ScriptLocation' => '<string>',
],
'Connections' => [
'Connections' => ['<string>', ...],
],
'DefaultArguments' => ['<string>', ...],
'Description' => '<string>',
'ExecutionProperty' => [
'MaxConcurrentRuns' => <integer>,
],
'GlueVersion' => '<string>',
'LogUri' => '<string>',
'MaxCapacity' => <float>,
'MaxRetries' => <integer>,
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'NumberOfWorkers' => <integer>,
'Role' => '<string>',
'SecurityConfiguration' => '<string>',
'Timeout' => <integer>,
'WorkerType' => 'Standard|G.1X|G.2X',
],
]);
Parameter Details
Members
- JobName
-
- Required: Yes
- Type: string
The name of the job definition to update.
- JobUpdate
-
- Required: Yes
- Type: JobUpdate structure
Specifies the values with which to update the job definition.
Result Syntax
[
'JobName' => '<string>',
]
Result Details
Errors
-
The input provided was not valid.
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
-
ConcurrentModificationException:
Two processes are trying to modify a resource simultaneously.
UpdateMLTransform
$result = $client->updateMLTransform([/* ... */]); $promise = $client->updateMLTransformAsync([/* ... */]);
Updates an existing machine learning transform. Call this operation to tune the algorithm parameters to achieve better results.
After calling this operation, you can call the StartMLEvaluationTaskRun operation to assess how well your new parameters achieved your goals (such as improving the quality of your machine learning transform, or making it more cost-effective).
Parameter Syntax
$result = $client->updateMLTransform([
'Description' => '<string>',
'MaxCapacity' => <float>,
'MaxRetries' => <integer>,
'Name' => '<string>',
'NumberOfWorkers' => <integer>,
'Parameters' => [
'FindMatchesParameters' => [
'AccuracyCostTradeoff' => <float>,
'EnforceProvidedLabels' => true || false,
'PrecisionRecallTradeoff' => <float>,
'PrimaryKeyColumnName' => '<string>',
],
'TransformType' => 'FIND_MATCHES', // REQUIRED
],
'Role' => '<string>',
'Timeout' => <integer>,
'TransformId' => '<string>', // REQUIRED
'WorkerType' => 'Standard|G.1X|G.2X',
]);
Parameter Details
Members
- Description
-
- Type: string
A description of the transform. The default is an empty string.
- MaxCapacity
-
- Type: double
The number of AWS Glue data processing units (DPUs) that are allocated to task runs for this transform. You can allocate from 2 to 100 DPUs; the default is 10. A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB of memory. For more information, see the AWS Glue pricing page.
When the
WorkerTypefield is set to a value other thanStandard, theMaxCapacityfield is set automatically and becomes read-only. - MaxRetries
-
- Type: int
The maximum number of times to retry a task for this transform after a task run fails.
- Name
-
- Type: string
The unique name that you gave the transform when you created it.
- NumberOfWorkers
-
- Type: int
The number of workers of a defined
workerTypethat are allocated when this task runs. - Parameters
-
- Type: TransformParameters structure
The configuration parameters that are specific to the transform type (algorithm) used. Conditionally dependent on the transform type.
- Role
-
- Type: string
The name or Amazon Resource Name (ARN) of the IAM role with the required permissions.
- Timeout
-
- Type: int
The timeout for a task run for this transform in minutes. This is the maximum time that a task run for this transform can consume resources before it is terminated and enters
TIMEOUTstatus. The default is 2,880 minutes (48 hours). - TransformId
-
- Required: Yes
- Type: string
A unique identifier that was generated when the transform was created.
- WorkerType
-
- Type: string
The type of predefined worker that is allocated when this task runs. Accepts a value of Standard, G.1X, or G.2X.
-
For the
Standardworker type, each worker provides 4 vCPU, 16 GB of memory and a 50GB disk, and 2 executors per worker. -
For the
G.1Xworker type, each worker provides 4 vCPU, 16 GB of memory and a 64GB disk, and 1 executor per worker. -
For the
G.2Xworker type, each worker provides 8 vCPU, 32 GB of memory and a 128GB disk, and 1 executor per worker.
Result Syntax
[
'TransformId' => '<string>',
]
Result Details
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
The operation timed out.
-
An internal service error occurred.
-
Access to a resource was denied.
UpdatePartition
$result = $client->updatePartition([/* ... */]); $promise = $client->updatePartitionAsync([/* ... */]);
Updates a partition.
Parameter Syntax
$result = $client->updatePartition([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'PartitionInput' => [ // REQUIRED
'LastAccessTime' => <integer || string || DateTime>,
'LastAnalyzedTime' => <integer || string || DateTime>,
'Parameters' => ['<string>', ...],
'StorageDescriptor' => [
'BucketColumns' => ['<string>', ...],
'Columns' => [
[
'Comment' => '<string>',
'Name' => '<string>', // REQUIRED
'Parameters' => ['<string>', ...],
'Type' => '<string>',
],
// ...
],
'Compressed' => true || false,
'InputFormat' => '<string>',
'Location' => '<string>',
'NumberOfBuckets' => <integer>,
'OutputFormat' => '<string>',
'Parameters' => ['<string>', ...],
'SerdeInfo' => [
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'SerializationLibrary' => '<string>',
],
'SkewedInfo' => [
'SkewedColumnNames' => ['<string>', ...],
'SkewedColumnValueLocationMaps' => ['<string>', ...],
'SkewedColumnValues' => ['<string>', ...],
],
'SortColumns' => [
[
'Column' => '<string>', // REQUIRED
'SortOrder' => <integer>, // REQUIRED
],
// ...
],
'StoredAsSubDirectories' => true || false,
],
'Values' => ['<string>', ...],
],
'PartitionValueList' => ['<string>', ...], // REQUIRED
'TableName' => '<string>', // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog where the partition to be updated resides. If none is provided, the AWS account ID is used by default.
- DatabaseName
-
- Required: Yes
- Type: string
The name of the catalog database in which the table in question resides.
- PartitionInput
-
- Required: Yes
- Type: PartitionInput structure
The new partition object to update the partition to.
- PartitionValueList
-
- Required: Yes
- Type: Array of strings
A list of the values defining the partition.
- TableName
-
- Required: Yes
- Type: string
The name of the table in which the partition to be updated is located.
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
-
An encryption operation failed.
UpdateTable
$result = $client->updateTable([/* ... */]); $promise = $client->updateTableAsync([/* ... */]);
Updates a metadata table in the Data Catalog.
Parameter Syntax
$result = $client->updateTable([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'SkipArchive' => true || false,
'TableInput' => [ // REQUIRED
'Description' => '<string>',
'LastAccessTime' => <integer || string || DateTime>,
'LastAnalyzedTime' => <integer || string || DateTime>,
'Name' => '<string>', // REQUIRED
'Owner' => '<string>',
'Parameters' => ['<string>', ...],
'PartitionKeys' => [
[
'Comment' => '<string>',
'Name' => '<string>', // REQUIRED
'Parameters' => ['<string>', ...],
'Type' => '<string>',
],
// ...
],
'Retention' => <integer>,
'StorageDescriptor' => [
'BucketColumns' => ['<string>', ...],
'Columns' => [
[
'Comment' => '<string>',
'Name' => '<string>', // REQUIRED
'Parameters' => ['<string>', ...],
'Type' => '<string>',
],
// ...
],
'Compressed' => true || false,
'InputFormat' => '<string>',
'Location' => '<string>',
'NumberOfBuckets' => <integer>,
'OutputFormat' => '<string>',
'Parameters' => ['<string>', ...],
'SerdeInfo' => [
'Name' => '<string>',
'Parameters' => ['<string>', ...],
'SerializationLibrary' => '<string>',
],
'SkewedInfo' => [
'SkewedColumnNames' => ['<string>', ...],
'SkewedColumnValueLocationMaps' => ['<string>', ...],
'SkewedColumnValues' => ['<string>', ...],
],
'SortColumns' => [
[
'Column' => '<string>', // REQUIRED
'SortOrder' => <integer>, // REQUIRED
],
// ...
],
'StoredAsSubDirectories' => true || false,
],
'TableType' => '<string>',
'ViewExpandedText' => '<string>',
'ViewOriginalText' => '<string>',
],
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog where the table resides. If none is provided, the AWS account ID is used by default.
- DatabaseName
-
- Required: Yes
- Type: string
The name of the catalog database in which the table resides. For Hive compatibility, this name is entirely lowercase.
- SkipArchive
-
- Type: boolean
By default,
UpdateTablealways creates an archived version of the table before updating it. However, ifskipArchiveis set to true,UpdateTabledoes not create the archived version. - TableInput
-
- Required: Yes
- Type: TableInput structure
An updated
TableInputobject to define the metadata table in the catalog.
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
-
ConcurrentModificationException:
Two processes are trying to modify a resource simultaneously.
-
ResourceNumberLimitExceededException:
A resource numerical limit was exceeded.
-
An encryption operation failed.
UpdateTrigger
$result = $client->updateTrigger([/* ... */]); $promise = $client->updateTriggerAsync([/* ... */]);
Updates a trigger definition.
Parameter Syntax
$result = $client->updateTrigger([
'Name' => '<string>', // REQUIRED
'TriggerUpdate' => [ // REQUIRED
'Actions' => [
[
'Arguments' => ['<string>', ...],
'CrawlerName' => '<string>',
'JobName' => '<string>',
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'SecurityConfiguration' => '<string>',
'Timeout' => <integer>,
],
// ...
],
'Description' => '<string>',
'Name' => '<string>',
'Predicate' => [
'Conditions' => [
[
'CrawlState' => 'RUNNING|SUCCEEDED|CANCELLED|FAILED',
'CrawlerName' => '<string>',
'JobName' => '<string>',
'LogicalOperator' => 'EQUALS',
'State' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
],
// ...
],
'Logical' => 'AND|ANY',
],
'Schedule' => '<string>',
],
]);
Parameter Details
Members
- Name
-
- Required: Yes
- Type: string
The name of the trigger to update.
- TriggerUpdate
-
- Required: Yes
- Type: TriggerUpdate structure
The new values with which to update the trigger.
Result Syntax
[
'Trigger' => [
'Actions' => [
[
'Arguments' => ['<string>', ...],
'CrawlerName' => '<string>',
'JobName' => '<string>',
'NotificationProperty' => [
'NotifyDelayAfter' => <integer>,
],
'SecurityConfiguration' => '<string>',
'Timeout' => <integer>,
],
// ...
],
'Description' => '<string>',
'Id' => '<string>',
'Name' => '<string>',
'Predicate' => [
'Conditions' => [
[
'CrawlState' => 'RUNNING|SUCCEEDED|CANCELLED|FAILED',
'CrawlerName' => '<string>',
'JobName' => '<string>',
'LogicalOperator' => 'EQUALS',
'State' => 'STARTING|RUNNING|STOPPING|STOPPED|SUCCEEDED|FAILED|TIMEOUT',
],
// ...
],
'Logical' => 'AND|ANY',
],
'Schedule' => '<string>',
'State' => 'CREATING|CREATED|ACTIVATING|ACTIVATED|DEACTIVATING|DEACTIVATED|DELETING|UPDATING',
'Type' => 'SCHEDULED|CONDITIONAL|ON_DEMAND',
'WorkflowName' => '<string>',
],
]
Result Details
Members
- Trigger
-
- Type: Trigger structure
The resulting trigger definition.
Errors
-
The input provided was not valid.
-
An internal service error occurred.
-
A specified entity does not exist
-
The operation timed out.
-
ConcurrentModificationException:
Two processes are trying to modify a resource simultaneously.
UpdateUserDefinedFunction
$result = $client->updateUserDefinedFunction([/* ... */]); $promise = $client->updateUserDefinedFunctionAsync([/* ... */]);
Updates an existing function definition in the Data Catalog.
Parameter Syntax
$result = $client->updateUserDefinedFunction([
'CatalogId' => '<string>',
'DatabaseName' => '<string>', // REQUIRED
'FunctionInput' => [ // REQUIRED
'ClassName' => '<string>',
'FunctionName' => '<string>',
'OwnerName' => '<string>',
'OwnerType' => 'USER|ROLE|GROUP',
'ResourceUris' => [
[
'ResourceType' => 'JAR|FILE|ARCHIVE',
'Uri' => '<string>',
],
// ...
],
],
'FunctionName' => '<string>', // REQUIRED
]);
Parameter Details
Members
- CatalogId
-
- Type: string
The ID of the Data Catalog where the function to be updated is located. If none is provided, the AWS account ID is used by default.
- DatabaseName
-
- Required: Yes
- Type: string
The name of the catalog database where the function to be updated is located.
- FunctionInput
-
- Required: Yes
- Type: UserDefinedFunctionInput structure
A
FunctionInputobject that redefines the function in the Data Catalog. - FunctionName
-
- Required: Yes
- Type: string
The name of the function.
Result Syntax
[]
Result Details
Errors
-
A specified entity does not exist
-
The input provided was not valid.
-
An internal service error occurred.
-
The operation timed out.
-
An encryption operation failed.
UpdateWorkflow
$result = $client->updateWorkflow([/* ... */]); $promise = $client->updateWorkflowAsync([/* ... */]);
Updates an existing workflow.
Parameter Syntax
$result = $client->updateWorkflow([
'DefaultRunProperties' => ['<string>', ...],
'Description' => '<string>',
'Name' => '<string>', // REQUIRED
]);
Parameter Details
Members
Result Syntax
[
'Name' => '<string>',
]
Result Details
Errors
-
The input provided was not valid.
-
A specified entity does not exist
-
An internal service error occurred.
-
The operation timed out.
-
ConcurrentModificationException:
Two processes are trying to modify a resource simultaneously.
Shapes
AccessDeniedException
Description
Access to a resource was denied.
Members
Action
Description
Defines an action to be initiated by a trigger.
Members
- Arguments
-
- Type: Associative array of custom strings keys (GenericString) to strings
The job arguments used when this trigger fires. For this job run, they replace the default arguments set in the job definition itself.
You can specify arguments here that your own job-execution script consumes, as well as arguments that AWS Glue itself consumes.
For information about how to specify and consume your own Job arguments, see the Calling AWS Glue APIs in Python topic in the developer guide.
For information about the key-value pairs that AWS Glue consumes to set up your job, see the Special Parameters Used by AWS Glue topic in the developer guide.
- CrawlerName
-
- Type: string
The name of the crawler to be used with this action.
- JobName
-
- Type: string
The name of a job to be executed.
- NotificationProperty
-
- Type: NotificationProperty structure
Specifies configuration properties of a job run notification.
- SecurityConfiguration
-
- Type: string
The name of the
SecurityConfigurationstructure to be used with this action. - Timeout
-
- Type: int
The
JobRuntimeout in minutes. This is the maximum time that a job run can consume resources before it is terminated and entersTIMEOUTstatus. The default is 2,880 minutes (48 hours). This overrides the timeout value set in the parent job.
AlreadyExistsException
Description
A resource to be created or added already exists.
Members
BatchStopJobRunError
Description
Records an error that occurred when attempting to stop a specified job run.
Members
- ErrorDetail
-
- Type: ErrorDetail structure
Specifies details about the error that was encountered.
- JobName
-
- Type: string
The name of the job definition that is used in the job run in question.
- JobRunId
-
- Type: string
The
JobRunIdof the job run in question.
BatchStopJobRunSuccessfulSubmission
Description
Records a successful request to stop a specified JobRun.
Members
CatalogEntry
Description
Specifies a table definition in the AWS Glue Data Catalog.
Members
CatalogImportStatus
Description
A structure containing migration status information.
Members
CatalogTarget
Description
Specifies an AWS Glue Data Catalog target.
Members
Classifier
Description
Classifiers are triggered during a crawl task. A classifier checks whether a given file is in a format it can handle. If it is, the classifier creates a schema in the form of a StructType object that matches that data format.
You can use the standard classifiers that AWS Glue provides, or you can write your own classifiers to best categorize your data sources and specify the appropriate schemas to use for them. A classifier can be a grok classifier, an XML classifier, a JSON classifier, or a custom CSV classifier, as specified in one of the fields in the Classifier object.
Members
- CsvClassifier
-
- Type: CsvClassifier structure
A classifier for comma-separated values (CSV).
- GrokClassifier
-
- Type: GrokClassifier structure
A classifier that uses
grok. - JsonClassifier
-
- Type: JsonClassifier structure
A classifier for JSON content.
- XMLClassifier
-
- Type: XMLClassifier structure
A classifier for XML content.
CloudWatchEncryption
Description
Specifies how Amazon CloudWatch data should be encrypted.
Members
CodeGenEdge
Description
Represents a directional edge in a directed acyclic graph (DAG).
Members
CodeGenNode
Description
Represents a node in a directed acyclic graph (DAG)
Members
- Args
-
- Required: Yes
- Type: Array of CodeGenNodeArg structures
Properties of the node, in the form of name-value pairs.
- Id
-
- Required: Yes
- Type: string
A node identifier that is unique within the node's graph.
- LineNumber
-
- Type: int
The line number of the node.
- NodeType
-
- Required: Yes
- Type: string
The type of node that this is.
CodeGenNodeArg
Description
An argument or property of a node.
Members
Column
Description
A column in a Table.
Members
ConcurrentModificationException
Description
Two processes are trying to modify a resource simultaneously.
Members
ConcurrentRunsExceededException
Description
Too many jobs are being run concurrently.
Members
Condition
Description
Defines a condition under which a trigger fires.
Members
- CrawlState
-
- Type: string
The state of the crawler to which this condition applies.
- CrawlerName
-
- Type: string
The name of the crawler to which this condition applies.
- JobName
-
- Type: string
The name of the job whose
JobRunsthis condition applies to, and on which this trigger waits. - LogicalOperator
-
- Type: string
A logical operator.
- State
-
- Type: string
The condition state. Currently, the values supported are
SUCCEEDED,STOPPED,TIMEOUT, andFAILED.
ConditionCheckFailureException
Description
A specified condition was not satisfied.
Members
ConfusionMatrix
Description
The confusion matrix shows you what your transform is predicting accurately and what types of errors it is making.
For more information, see Confusion matrix in Wikipedia.
Members
- NumFalseNegatives
-
- Type: long (int|float)
The number of matches in the data that the transform didn't find, in the confusion matrix for your transform.
- NumFalsePositives
-
- Type: long (int|float)
The number of nonmatches in the data that the transform incorrectly classified as a match, in the confusion matrix for your transform.
- NumTrueNegatives
-
- Type: long (int|float)
The number of nonmatches in the data that the transform correctly rejected, in the confusion matrix for your transform.
- NumTruePositives
-
- Type: long (int|float)
The number of matches in the data that the transform correctly found, in the confusion matrix for your transform.
Connection
Description
Defines a connection to a data source.
Members
- ConnectionProperties
-
- Type: Associative array of custom strings keys (ConnectionPropertyKey) to strings
These key-value pairs define parameters for the connection:
-
HOST- The host URI: either the fully qualified domain name (FQDN) or the IPv4 address of the database host. -
PORT- The port number, between 1024 and 65535, of the port on which the database host is listening for database connections. -
USER_NAME- The name under which to log in to the database. The value string forUSER_NAMEis "USERNAME". -
PASSWORD- A password, if one is used, for the user name. -
ENCRYPTED_PASSWORD- When you enable connection password protection by settingConnectionPasswordEncryptionin the Data Catalog encryption settings, this field stores the encrypted password. -
JDBC_DRIVER_JAR_URI- The Amazon Simple Storage Service (Amazon S3) path of the JAR file that contains the JDBC driver to use. -
JDBC_DRIVER_CLASS_NAME- The class name of the JDBC driver to use. -
JDBC_ENGINE- The name of the JDBC engine to use. -
JDBC_ENGINE_VERSION- The version of the JDBC engine to use. -
CONFIG_FILES- (Reserved for future use.) -
INSTANCE_ID- The instance ID to use. -
JDBC_CONNECTION_URL- The URL for the JDBC connection. -
JDBC_ENFORCE_SSL- A Boolean string (true, false) specifying whether Secure Sockets Layer (SSL) with hostname matching is enforced for the JDBC connection on the client. The default is false.
- ConnectionType
-
- Type: string
The type of the connection. Currently, only JDBC is supported; SFTP is not supported.
- CreationTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The time that this connection definition was created.
- Description
-
- Type: string
The description of the connection.
- LastUpdatedBy
-
- Type: string
The user, group, or role that last updated this connection definition.
- LastUpdatedTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The last time that this connection definition was updated.
- MatchCriteria
-
- Type: Array of strings
A list of criteria that can be used in selecting this connection.
- Name
-
- Type: string
The name of the connection definition.
- PhysicalConnectionRequirements
-
- Type: PhysicalConnectionRequirements structure
A map of physical connection requirements, such as virtual private cloud (VPC) and
SecurityGroup, that are needed to make this connection successfully.
ConnectionInput
Description
A structure that is used to specify a connection to create or update.
Members
- ConnectionProperties
-
- Required: Yes
- Type: Associative array of custom strings keys (ConnectionPropertyKey) to strings
These key-value pairs define parameters for the connection.
- ConnectionType
-
- Required: Yes
- Type: string
The type of the connection. Currently, only JDBC is supported; SFTP is not supported.
- Description
-
- Type: string
The description of the connection.
- MatchCriteria
-
- Type: Array of strings
A list of criteria that can be used in selecting this connection.
- Name
-
- Required: Yes
- Type: string
The name of the connection.
- PhysicalConnectionRequirements
-
- Type: PhysicalConnectionRequirements structure
A map of physical connection requirements, such as virtual private cloud (VPC) and
SecurityGroup, that are needed to successfully make this connection.
ConnectionPasswordEncryption
Description
The data structure used by the Data Catalog to encrypt the password as part of CreateConnection or UpdateConnection and store it in the ENCRYPTED_PASSWORD field in the connection properties. You can enable catalog encryption or only password encryption.
When a CreationConnection request arrives containing a password, the Data Catalog first encrypts the password using your AWS KMS key. It then encrypts the whole connection object again if catalog encryption is also enabled.
This encryption requires that you set AWS KMS key permissions to enable or restrict access on the password key according to your security requirements. For example, you might want only administrators to have decrypt permission on the password key.
Members
- AwsKmsKeyId
-
- Type: string
An AWS KMS key that is used to encrypt the connection password.
If connection password protection is enabled, the caller of
CreateConnectionandUpdateConnectionneeds at leastkms:Encryptpermission on the specified AWS KMS key, to encrypt passwords before storing them in the Data Catalog.You can set the decrypt permission to enable or restrict access on the password key according to your security requirements.
- ReturnConnectionPasswordEncrypted
-
- Required: Yes
- Type: boolean
When the
ReturnConnectionPasswordEncryptedflag is set to "true", passwords remain encrypted in the responses ofGetConnectionandGetConnections. This encryption takes effect independently from catalog encryption.
ConnectionsList
Description
Specifies the connections used by a job.
Members
Crawl
Description
The details of a crawl in the workflow.
Members
- CompletedOn
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The date and time on which the crawl completed.
- ErrorMessage
-
- Type: string
The error message associated with the crawl.
- LogGroup
-
- Type: string
The log group associated with the crawl.
- LogStream
-
- Type: string
The log stream associated with the crawl.
- StartedOn
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The date and time on which the crawl started.
- State
-
- Type: string
The state of the crawler.
Crawler
Description
Specifies a crawler program that examines a data source and uses classifiers to try to determine its schema. If successful, the crawler records metadata concerning the data source in the AWS Glue Data Catalog.
Members
- Classifiers
-
- Type: Array of strings
A list of UTF-8 strings that specify the custom classifiers that are associated with the crawler.
- Configuration
-
- Type: string
Crawler configuration information. This versioned JSON string allows users to specify aspects of a crawler's behavior. For more information, see Configuring a Crawler.
- CrawlElapsedTime
-
- Type: long (int|float)
If the crawler is running, contains the total time elapsed since the last crawl began.
- CrawlerSecurityConfiguration
-
- Type: string
The name of the
SecurityConfigurationstructure to be used by this crawler. - CreationTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The time that the crawler was created.
- DatabaseName
-
- Type: string
The name of the database in which the crawler's output is stored.
- Description
-
- Type: string
A description of the crawler.
- LastCrawl
-
- Type: LastCrawlInfo structure
The status of the last crawl, and potentially error information if an error occurred.
- LastUpdated
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The time that the crawler was last updated.
- Name
-
- Type: string
The name of the crawler.
- Role
-
- Type: string
The Amazon Resource Name (ARN) of an IAM role that's used to access customer resources, such as Amazon Simple Storage Service (Amazon S3) data.
- Schedule
-
- Type: Schedule structure
For scheduled crawlers, the schedule when the crawler runs.
- SchemaChangePolicy
-
- Type: SchemaChangePolicy structure
The policy that specifies update and delete behaviors for the crawler.
- State
-
- Type: string
Indicates whether the crawler is running, or whether a run is pending.
- TablePrefix
-
- Type: string
The prefix added to the names of tables that are created.
- Targets
-
- Type: CrawlerTargets structure
A collection of targets to crawl.
- Version
-
- Type: long (int|float)
The version of the crawler.
CrawlerMetrics
Description
Metrics for a specified crawler.
Members
- CrawlerName
-
- Type: string
The name of the crawler.
- LastRuntimeSeconds
-
- Type: double
The duration of the crawler's most recent run, in seconds.
- MedianRuntimeSeconds
-
- Type: double
The median duration of this crawler's runs, in seconds.
- StillEstimating
-
- Type: boolean
True if the crawler is still estimating how long it will take to complete this run.
- TablesCreated
-
- Type: int
The number of tables created by this crawler.
- TablesDeleted
-
- Type: int
The number of tables deleted by this crawler.
- TablesUpdated
-
- Type: int
The number of tables updated by this crawler.
- TimeLeftSeconds
-
- Type: double
The estimated time left to complete a running crawl.
CrawlerNodeDetails
Description
The details of a Crawler node present in the workflow.
Members
- Crawls
-
- Type: Array of Crawl structures
A list of crawls represented by the crawl node.
CrawlerNotRunningException
Description
The specified crawler is not running.
Members
CrawlerRunningException
Description
The operation cannot be performed because the crawler is already running.
Members
CrawlerStoppingException
Description
The specified crawler is stopping.
Members
CrawlerTargets
Description
Specifies data stores to crawl.
Members
- CatalogTargets
-
- Type: Array of CatalogTarget structures
Specifies AWS Glue Data Catalog targets.
- DynamoDBTargets
-
- Type: Array of DynamoDBTarget structures
Specifies Amazon DynamoDB targets.
- JdbcTargets
-
- Type: Array of JdbcTarget structures
Specifies JDBC targets.
- S3Targets
-
- Type: Array of S3Target structures
Specifies Amazon Simple Storage Service (Amazon S3) targets.
CreateCsvClassifierRequest
Description
Specifies a custom CSV classifier for CreateClassifier to create.
Members
- AllowSingleColumn
-
- Type: boolean
Enables the processing of files that contain only one column.
- ContainsHeader
-
- Type: string
Indicates whether the CSV file contains a header.
- Delimiter
-
- Type: string
A custom symbol to denote what separates each column entry in the row.
- DisableValueTrimming
-
- Type: boolean
Specifies not to trim values before identifying the type of column values. The default value is true.
- Header
-
- Type: Array of strings
A list of strings representing column names.
- Name
-
- Required: Yes
- Type: string
The name of the classifier.
- QuoteSymbol
-
- Type: string
A custom symbol to denote what combines content into a single column value. Must be different from the column delimiter.
CreateGrokClassifierRequest
Description
Specifies a grok classifier for CreateClassifier to create.
Members
- Classification
-
- Required: Yes
- Type: string
An identifier of the data format that the classifier matches, such as Twitter, JSON, Omniture logs, Amazon CloudWatch Logs, and so on.
- CustomPatterns
-
- Type: string
Optional custom grok patterns used by this classifier.
- GrokPattern
-
- Required: Yes
- Type: string
The grok pattern used by this classifier.
- Name
-
- Required: Yes
- Type: string
The name of the new classifier.
CreateJsonClassifierRequest
Description
Specifies a JSON classifier for CreateClassifier to create.
Members
- JsonPath
-
- Required: Yes
- Type: string
A
JsonPathstring defining the JSON data for the classifier to classify. AWS Glue supports a subset ofJsonPath, as described in Writing JsonPath Custom Classifiers. - Name
-
- Required: Yes
- Type: string
The name of the classifier.
CreateXMLClassifierRequest
Description
Specifies an XML classifier for CreateClassifier to create.
Members
- Classification
-
- Required: Yes
- Type: string
An identifier of the data format that the classifier matches.
- Name
-
- Required: Yes
- Type: string
The name of the classifier.
- RowTag
-
- Type: string
The XML tag designating the element that contains each record in an XML document being parsed. This can't identify a self-closing element (closed by
/>). An empty row element that contains only attributes can be parsed as long as it ends with a closing tag (for example,<row item_a="A" item_b="B"></row>is okay, but<row item_a="A" item_b="B" />is not).
CsvClassifier
Description
A classifier for custom CSV content.
Members
- AllowSingleColumn
-
- Type: boolean
Enables the processing of files that contain only one column.
- ContainsHeader
-
- Type: string
Indicates whether the CSV file contains a header.
- CreationTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The time that this classifier was registered.
- Delimiter
-
- Type: string
A custom symbol to denote what separates each column entry in the row.
- DisableValueTrimming
-
- Type: boolean
Specifies not to trim values before identifying the type of column values. The default value is
true. - Header
-
- Type: Array of strings
A list of strings representing column names.
- LastUpdated
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The time that this classifier was last updated.
- Name
-
- Required: Yes
- Type: string
The name of the classifier.
- QuoteSymbol
-
- Type: string
A custom symbol to denote what combines content into a single column value. It must be different from the column delimiter.
- Version
-
- Type: long (int|float)
The version of this classifier.
DataCatalogEncryptionSettings
Description
Contains configuration information for maintaining Data Catalog security.
Members
- ConnectionPasswordEncryption
-
- Type: ConnectionPasswordEncryption structure
When connection password protection is enabled, the Data Catalog uses a customer-provided key to encrypt the password as part of
CreateConnectionorUpdateConnectionand store it in theENCRYPTED_PASSWORDfield in the connection properties. You can enable catalog encryption or only password encryption. - EncryptionAtRest
-
- Type: EncryptionAtRest structure
Specifies the encryption-at-rest configuration for the Data Catalog.
DataLakePrincipal
Description
The AWS Lake Formation principal.
Members
Database
Description
The Database object represents a logical grouping of tables that might reside in a Hive metastore or an RDBMS.
Members
- CreateTableDefaultPermissions
-
- Type: Array of PrincipalPermissions structures
Creates a set of default permissions on the table for principals.
- CreateTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The time at which the metadata database was created in the catalog.
- Description
-
- Type: string
A description of the database.
- LocationUri
-
- Type: string
The location of the database (for example, an HDFS path).
- Name
-
- Required: Yes
- Type: string
The name of the database. For Hive compatibility, this is folded to lowercase when it is stored.
- Parameters
-
- Type: Associative array of custom strings keys (KeyString) to strings
These key-value pairs define parameters and properties of the database.
DatabaseInput
Description
The structure used to create or update a database.
Members
- CreateTableDefaultPermissions
-
- Type: Array of PrincipalPermissions structures
Creates a set of default permissions on the table for principals.
- Description
-
- Type: string
A description of the database.
- LocationUri
-
- Type: string
The location of the database (for example, an HDFS path).
- Name
-
- Required: Yes
- Type: string
The name of the database. For Hive compatibility, this is folded to lowercase when it is stored.
- Parameters
-
- Type: Associative array of custom strings keys (KeyString) to strings
These key-value pairs define parameters and properties of the database.
These key-value pairs define parameters and properties of the database.
DevEndpoint
Description
A development endpoint where a developer can remotely debug extract, transform, and load (ETL) scripts.
Members
- Arguments
-
- Type: Associative array of custom strings keys (GenericString) to strings
A map of arguments used to configure the
DevEndpoint.Currently, only
"--enable-glue-datacatalog": ""is supported as a valid argument. - AvailabilityZone
-
- Type: string
The AWS Availability Zone where this
DevEndpointis located. - CreatedTimestamp
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The point in time at which this DevEndpoint was created.
- EndpointName
-
- Type: string
The name of the
DevEndpoint. - ExtraJarsS3Path
-
- Type: string
The path to one or more Java
.jarfiles in an S3 bucket that should be loaded in yourDevEndpoint.You can only use pure Java/Scala libraries with a
DevEndpoint. - ExtraPythonLibsS3Path
-
- Type: string
The paths to one or more Python libraries in an Amazon S3 bucket that should be loaded in your
DevEndpoint. Multiple values must be complete paths separated by a comma.You can only use pure Python libraries with a
DevEndpoint. Libraries that rely on C extensions, such as the pandas Python data analysis library, are not currently supported. - FailureReason
-
- Type: string
The reason for a current failure in this
DevEndpoint. - LastModifiedTimestamp
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The point in time at which this
DevEndpointwas last modified. - LastUpdateStatus
-
- Type: string
The status of the last update.
- NumberOfNodes
-
- Type: int
The number of AWS Glue Data Processing Units (DPUs) allocated to this
DevEndpoint. - NumberOfWorkers
-
- Type: int
The number of workers of a defined
workerTypethat are allocated to the development endpoint.The maximum number of workers you can define are 299 for
G.1X, and 149 forG.2X. - PrivateAddress
-
- Type: string
A private IP address to access the
DevEndpointwithin a VPC if theDevEndpointis created within one. ThePrivateAddressfield is present only when you create theDevEndpointwithin your VPC. - PublicAddress
-
- Type: string
The public IP address used by this
DevEndpoint. ThePublicAddressfield is present only when you create a non-virtual private cloud (VPC)DevEndpoint. - PublicKey
-
- Type: string
The public key to be used by this
DevEndpointfor authentication. This attribute is provided for backward compatibility because the recommended attribute to use is public keys. - PublicKeys
-
- Type: Array of strings
A list of public keys to be used by the
DevEndpointsfor authentication. Using this attribute is preferred over a single public key because the public keys allow you to have a different private key per client.If you previously created an endpoint with a public key, you must remove that key to be able to set a list of public keys. Call the
UpdateDevEndpointAPI operation with the public key content in thedeletePublicKeysattribute, and the list of new keys in theaddPublicKeysattribute. - RoleArn
-
- Type: string
The Amazon Resource Name (ARN) of the IAM role used in this
DevEndpoint. - SecurityConfiguration
-
- Type: string
The name of the
SecurityConfigurationstructure to be used with thisDevEndpoint. - SecurityGroupIds
-
- Type: Array of strings
A list of security group identifiers used in this
DevEndpoint. - Status
-
- Type: string
The current status of this
DevEndpoint. - SubnetId
-
- Type: string
The subnet ID for this
DevEndpoint. - VpcId
-
- Type: string
The ID of the virtual private cloud (VPC) used by this
DevEndpoint. - WorkerType
-
- Type: string
The type of predefined worker that is allocated to the development endpoint. Accepts a value of Standard, G.1X, or G.2X.
-
For the
Standardworker type, each worker provides 4 vCPU, 16 GB of memory and a 50GB disk, and 2 executors per worker. -
For the
G.1Xworker type, each worker maps to 1 DPU (4 vCPU, 16 GB of memory, 64 GB disk), and provides 1 executor per worker. We recommend this worker type for memory-intensive jobs. -
For the
G.2Xworker type, each worker maps to 2 DPU (8 vCPU, 32 GB of memory, 128 GB disk), and provides 1 executor per worker. We recommend this worker type for memory-intensive jobs.
Known issue: when a development endpoint is created with the
G.2XWorkerTypeconfiguration, the Spark drivers for the development endpoint will run on 4 vCPU, 16 GB of memory, and a 64 GB disk. - YarnEndpointAddress
-
- Type: string
The YARN endpoint address used by this
DevEndpoint. - ZeppelinRemoteSparkInterpreterPort
-
- Type: int
The Apache Zeppelin port for the remote Apache Spark interpreter.
DevEndpointCustomLibraries
Description
Custom libraries to be loaded into a development endpoint.
Members
- ExtraJarsS3Path
-
- Type: string
The path to one or more Java
.jarfiles in an S3 bucket that should be loaded in yourDevEndpoint.You can only use pure Java/Scala libraries with a
DevEndpoint. - ExtraPythonLibsS3Path
-
- Type: string
The paths to one or more Python libraries in an Amazon Simple Storage Service (Amazon S3) bucket that should be loaded in your
DevEndpoint. Multiple values must be complete paths separated by a comma.You can only use pure Python libraries with a
DevEndpoint. Libraries that rely on C extensions, such as the pandas Python data analysis library, are not currently supported.
DynamoDBTarget
Description
Specifies an Amazon DynamoDB table to crawl.
Members
Edge
Description
An edge represents a directed connection between two AWS Glue components which are part of the workflow the edge belongs to.
Members
EncryptionAtRest
Description
Specifies the encryption-at-rest configuration for the Data Catalog.
Members
EncryptionConfiguration
Description
Specifies an encryption configuration.
Members
- CloudWatchEncryption
-
- Type: CloudWatchEncryption structure
The encryption configuration for Amazon CloudWatch.
- JobBookmarksEncryption
-
- Type: JobBookmarksEncryption structure
The encryption configuration for job bookmarks.
- S3Encryption
-
- Type: Array of S3Encryption structures
The encryption configuration for Amazon Simple Storage Service (Amazon S3) data.
EntityNotFoundException
Description
A specified entity does not exist
Members
ErrorDetail
Description
Contains details about an error.
Members
EvaluationMetrics
Description
Evaluation metrics provide an estimate of the quality of your machine learning transform.
Members
- FindMatchesMetrics
-
- Type: FindMatchesMetrics structure
The evaluation metrics for the find matches algorithm.
- TransformType
-
- Required: Yes
- Type: string
The type of machine learning transform.
ExecutionProperty
Description
An execution property of a job.
Members
ExportLabelsTaskRunProperties
Description
Specifies configuration properties for an exporting labels task run.
Members
FindMatchesMetrics
Description
The evaluation metrics for the find matches algorithm. The quality of your machine learning transform is measured by getting your transform to predict some matches and comparing the results to known matches from the same dataset. The quality metrics are based on a subset of your data, so they are not precise.
Members
- AreaUnderPRCurve
-
- Type: double
The area under the precision/recall curve (AUPRC) is a single number measuring the overall quality of the transform, that is independent of the choice made for precision vs. recall. Higher values indicate that you have a more attractive precision vs. recall tradeoff.
For more information, see Precision and recall in Wikipedia.
- ConfusionMatrix
-
- Type: ConfusionMatrix structure
The confusion matrix shows you what your transform is predicting accurately and what types of errors it is making.
For more information, see Confusion matrix in Wikipedia.
- F1
-
- Type: double
The maximum F1 metric indicates the transform's accuracy between 0 and 1, where 1 is the best accuracy.
For more information, see F1 score in Wikipedia.
- Precision
-
- Type: double
The precision metric indicates when often your transform is correct when it predicts a match. Specifically, it measures how well the transform finds true positives from the total true positives possible.
For more information, see Precision and recall in Wikipedia.
- Recall
-
- Type: double
The recall metric indicates that for an actual match, how often your transform predicts the match. Specifically, it measures how well the transform finds true positives from the total records in the source data.
For more information, see Precision and recall in Wikipedia.
FindMatchesParameters
Description
The parameters to configure the find matches transform.
Members
- AccuracyCostTradeoff
-
- Type: double
The value that is selected when tuning your transform for a balance between accuracy and cost. A value of 0.5 means that the system balances accuracy and cost concerns. A value of 1.0 means a bias purely for accuracy, which typically results in a higher cost, sometimes substantially higher. A value of 0.0 means a bias purely for cost, which results in a less accurate
FindMatchestransform, sometimes with unacceptable accuracy.Accuracy measures how well the transform finds true positives and true negatives. Increasing accuracy requires more machine resources and cost. But it also results in increased recall.
Cost measures how many compute resources, and thus money, are consumed to run the transform.
- EnforceProvidedLabels
-
- Type: boolean
The value to switch on or off to force the output to match the provided labels from users. If the value is
True, thefind matchestransform forces the output to match the provided labels. The results override the normal conflation results. If the value isFalse, thefind matchestransform does not ensure all the labels provided are respected, and the results rely on the trained model.Note that setting this value to true may increase the conflation execution time.
- PrecisionRecallTradeoff
-
- Type: double
The value selected when tuning your transform for a balance between precision and recall. A value of 0.5 means no preference; a value of 1.0 means a bias purely for precision, and a value of 0.0 means a bias for recall. Because this is a tradeoff, choosing values close to 1.0 means very low recall, and choosing values close to 0.0 results in very low precision.
The precision metric indicates how often your model is correct when it predicts a match.
The recall metric indicates that for an actual match, how often your model predicts the match.
- PrimaryKeyColumnName
-
- Type: string
The name of a column that uniquely identifies rows in the source table. Used to help identify matching records.
FindMatchesTaskRunProperties
Description
Specifies configuration properties for a Find Matches task run.
Members
GetConnectionsFilter
Description
Filters the connection definitions that are returned by the GetConnections API operation.
Members
GlueEncryptionException
Description
An encryption operation failed.
Members
GlueTable
Description
The database and table in the AWS Glue Data Catalog that is used for input or output data.
Members
- CatalogId
-
- Type: string
A unique identifier for the AWS Glue Data Catalog.
- ConnectionName
-
- Type: string
The name of the connection to the AWS Glue Data Catalog.
- DatabaseName
-
- Required: Yes
- Type: string
A database name in the AWS Glue Data Catalog.
- TableName
-
- Required: Yes
- Type: string
A table name in the AWS Glue Data Catalog.
GrokClassifier
Description
A classifier that uses grok patterns.
Members
- Classification
-
- Required: Yes
- Type: string
An identifier of the data format that the classifier matches, such as Twitter, JSON, Omniture logs, and so on.
- CreationTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The time that this classifier was registered.
- CustomPatterns
-
- Type: string
Optional custom grok patterns defined by this classifier. For more information, see custom patterns in Writing Custom Classifiers.
- GrokPattern
-
- Required: Yes
- Type: string
The grok pattern applied to a data store by this classifier. For more information, see built-in patterns in Writing Custom Classifiers.
- LastUpdated
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The time that this classifier was last updated.
- Name
-
- Required: Yes
- Type: string
The name of the classifier.
- Version
-
- Type: long (int|float)
The version of this classifier.
IdempotentParameterMismatchException
Description
The same unique identifier was associated with two different records.
Members
ImportLabelsTaskRunProperties
Description
Specifies configuration properties for an importing labels task run.
Members
InternalServiceException
Description
An internal service error occurred.
Members
InvalidInputException
Description
The input provided was not valid.
Members
JdbcTarget
Description
Specifies a JDBC data store to crawl.
Members
- ConnectionName
-
- Type: string
The name of the connection to use to connect to the JDBC target.
- Exclusions
-
- Type: Array of strings
A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler.
- Path
-
- Type: string
The path of the JDBC target.
Job
Description
Specifies a job definition.
Members
- AllocatedCapacity
-
- Type: int
This field is deprecated. Use
MaxCapacityinstead.The number of AWS Glue data processing units (DPUs) allocated to runs of this job. You can allocate from 2 to 100 DPUs; the default is 10. A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB of memory. For more information, see the AWS Glue pricing page.
- Command
-
- Type: JobCommand structure
The
JobCommandthat executes this job. - Connections
-
- Type: ConnectionsList structure
The connections used for this job.
- CreatedOn
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The time and date that this job definition was created.
- DefaultArguments
-
- Type: Associative array of custom strings keys (GenericString) to strings
The default arguments for this job, specified as name-value pairs.
You can specify arguments here that your own job-execution script consumes, as well as arguments that AWS Glue itself consumes.
For information about how to specify and consume your own Job arguments, see the Calling AWS Glue APIs in Python topic in the developer guide.
For information about the key-value pairs that AWS Glue consumes to set up your job, see the Special Parameters Used by AWS Glue topic in the developer guide.
- Description
-
- Type: string
A description of the job.
- ExecutionProperty
-
- Type: ExecutionProperty structure
An
ExecutionPropertyspecifying the maximum number of concurrent runs allowed for this job. - GlueVersion
-
- Type: string
Glue version determines the versions of Apache Spark and Python that AWS Glue supports. The Python version indicates the version supported for jobs of type Spark.
For more information about the available AWS Glue versions and corresponding Spark and Python versions, see Glue version in the developer guide.
Jobs that are created without specifying a Glue version default to Glue 0.9.
- LastModifiedOn
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The last point in time when this job definition was modified.
- LogUri
-
- Type: string
This field is reserved for future use.
- MaxCapacity
-
- Type: double
The number of AWS Glue data processing units (DPUs) that can be allocated when this job runs. A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB of memory. For more information, see the AWS Glue pricing page.
Do not set
Max Capacityif usingWorkerTypeandNumberOfWorkers.The value that can be allocated for
MaxCapacitydepends on whether you are running a Python shell job or an Apache Spark ETL job:-
When you specify a Python shell job (
JobCommand.Name="pythonshell"), you can allocate either 0.0625 or 1 DPU. The default is 0.0625 DPU. -
When you specify an Apache Spark ETL job (
JobCommand.Name="glueetl"), you can allocate from 2 to 100 DPUs. The default is 10 DPUs. This job type cannot have a fractional DPU allocation.
- MaxRetries
-
- Type: int
The maximum number of times to retry this job after a JobRun fails.
- Name
-
- Type: string
The name you assign to this job definition.
- NotificationProperty
-
- Type: NotificationProperty structure
Specifies configuration properties of a job notification.
- NumberOfWorkers
-
- Type: int
The number of workers of a defined
workerTypethat are allocated when a job runs.The maximum number of workers you can define are 299 for
G.1X, and 149 forG.2X. - Role
-
- Type: string
The name or Amazon Resource Name (ARN) of the IAM role associated with this job.
- SecurityConfiguration
-
- Type: string
The name of the
SecurityConfigurationstructure to be used with this job. - Timeout
-
- Type: int
The job timeout in minutes. This is the maximum time that a job run can consume resources before it is terminated and enters
TIMEOUTstatus. The default is 2,880 minutes (48 hours). - WorkerType
-
- Type: string
The type of predefined worker that is allocated when a job runs. Accepts a value of Standard, G.1X, or G.2X.
-
For the
Standardworker type, each worker provides 4 vCPU, 16 GB of memory and a 50GB disk, and 2 executors per worker. -
For the
G.1Xworker type, each worker maps to 1 DPU (4 vCPU, 16 GB of memory, 64 GB disk), and provides 1 executor per worker. We recommend this worker type for memory-intensive jobs. -
For the
G.2Xworker type, each worker maps to 2 DPU (8 vCPU, 32 GB of memory, 128 GB disk), and provides 1 executor per worker. We recommend this worker type for memory-intensive jobs.
JobBookmarkEntry
Description
Defines a point that a job can resume processing.
Members
- Attempt
-
- Type: int
The attempt ID number.
- JobBookmark
-
- Type: string
The bookmark itself.
- JobName
-
- Type: string
The name of the job in question.
- PreviousRunId
-
- Type: string
The unique run identifier associated with the previous job run.
- Run
-
- Type: int
The run ID number.
- RunId
-
- Type: string
The run ID number.
- Version
-
- Type: int
The version of the job.
JobBookmarksEncryption
Description
Specifies how job bookmark data should be encrypted.
Members
JobCommand
Description
Specifies code executed when a job is run.
Members
- Name
-
- Type: string
The name of the job command. For an Apache Spark ETL job, this must be
glueetl. For a Python shell job, it must bepythonshell. - PythonVersion
-
- Type: string
The Python version being used to execute a Python shell job. Allowed values are 2 or 3.
- ScriptLocation
-
- Type: string
Specifies the Amazon Simple Storage Service (Amazon S3) path to a script that executes a job.
JobNodeDetails
Description
The details of a Job node present in the workflow.
Members
- JobRuns
-
- Type: Array of JobRun structures
The information for the job runs represented by the job node.
JobRun
Description
Contains information about a job run.
Members
- AllocatedCapacity
-
- Type: int
This field is deprecated. Use
MaxCapacityinstead.The number of AWS Glue data processing units (DPUs) allocated to this JobRun. From 2 to 100 DPUs can be allocated; the default is 10. A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB of memory. For more information, see the AWS Glue pricing page.
- Arguments
-
- Type: Associative array of custom strings keys (GenericString) to strings
The job arguments associated with this run. For this job run, they replace the default arguments set in the job definition itself.
You can specify arguments here that your own job-execution script consumes, as well as arguments that AWS Glue itself consumes.
For information about how to specify and consume your own job arguments, see the Calling AWS Glue APIs in Python topic in the developer guide.
For information about the key-value pairs that AWS Glue consumes to set up your job, see the Special Parameters Used by AWS Glue topic in the developer guide.
- Attempt
-
- Type: int
The number of the attempt to run this job.
- CompletedOn
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The date and time that this job run completed.
- ErrorMessage
-
- Type: string
An error message associated with this job run.
- ExecutionTime
-
- Type: int
The amount of time (in seconds) that the job run consumed resources.
- GlueVersion
-
- Type: string
Glue version determines the versions of Apache Spark and Python that AWS Glue supports. The Python version indicates the version supported for jobs of type Spark.
For more information about the available AWS Glue versions and corresponding Spark and Python versions, see Glue version in the developer guide.
Jobs that are created without specifying a Glue version default to Glue 0.9.
- Id
-
- Type: string
The ID of this job run.
- JobName
-
- Type: string
The name of the job definition being used in this run.
- JobRunState
-
- Type: string
The current state of the job run.
- LastModifiedOn
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The last time that this job run was modified.
- LogGroupName
-
- Type: string
The name of the log group for secure logging that can be server-side encrypted in Amazon CloudWatch using AWS KMS. This name can be
/aws-glue/jobs/, in which case the default encryption isNONE. If you add a role name andSecurityConfigurationname (in other words,/aws-glue/jobs-yourRoleName-yourSecurityConfigurationName/), then that security configuration is used to encrypt the log group. - MaxCapacity
-
- Type: double
The number of AWS Glue data processing units (DPUs) that can be allocated when this job runs. A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB of memory. For more information, see the AWS Glue pricing page.
Do not set
Max Capacityif usingWorkerTypeandNumberOfWorkers.The value that can be allocated for
MaxCapacitydepends on whether you are running a Python shell job or an Apache Spark ETL job:-
When you specify a Python shell job (
JobCommand.Name="pythonshell"), you can allocate either 0.0625 or 1 DPU. The default is 0.0625 DPU. -
When you specify an Apache Spark ETL job (
JobCommand.Name="glueetl"), you can allocate from 2 to 100 DPUs. The default is 10 DPUs. This job type cannot have a fractional DPU allocation.
- NotificationProperty
-
- Type: NotificationProperty structure
Specifies configuration properties of a job run notification.
- NumberOfWorkers
-
- Type: int
The number of workers of a defined
workerTypethat are allocated when a job runs.The maximum number of workers you can define are 299 for
G.1X, and 149 forG.2X. - PredecessorRuns
-
- Type: Array of Predecessor structures
A list of predecessors to this job run.
- PreviousRunId
-
- Type: string
The ID of the previous run of this job. For example, the
JobRunIdspecified in theStartJobRunaction. - SecurityConfiguration
-
- Type: string
The name of the
SecurityConfigurationstructure to be used with this job run. - StartedOn
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The date and time at which this job run was started.
- Timeout
-
- Type: int
The
JobRuntimeout in minutes. This is the maximum time that a job run can consume resources before it is terminated and entersTIMEOUTstatus. The default is 2,880 minutes (48 hours). This overrides the timeout value set in the parent job. - TriggerName
-
- Type: string
The name of the trigger that started this job run.
- WorkerType
-
- Type: string
The type of predefined worker that is allocated when a job runs. Accepts a value of Standard, G.1X, or G.2X.
-
For the
Standardworker type, each worker provides 4 vCPU, 16 GB of memory and a 50GB disk, and 2 executors per worker. -
For the
G.1Xworker type, each worker provides 4 vCPU, 16 GB of memory and a 64GB disk, and 1 executor per worker. -
For the
G.2Xworker type, each worker provides 8 vCPU, 32 GB of memory and a 128GB disk, and 1 executor per worker.
JobUpdate
Description
Specifies information used to update an existing job definition. The previous job definition is completely overwritten by this information.
Members
- AllocatedCapacity
-
- Type: int
This field is deprecated. Use
MaxCapacityinstead.The number of AWS Glue data processing units (DPUs) to allocate to this job. You can allocate from 2 to 100 DPUs; the default is 10. A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB of memory. For more information, see the AWS Glue pricing page.
- Command
-
- Type: JobCommand structure
The
JobCommandthat executes this job (required). - Connections
-
- Type: ConnectionsList structure
The connections used for this job.
- DefaultArguments
-
- Type: Associative array of custom strings keys (GenericString) to strings
The default arguments for this job.
You can specify arguments here that your own job-execution script consumes, as well as arguments that AWS Glue itself consumes.
For information about how to specify and consume your own Job arguments, see the Calling AWS Glue APIs in Python topic in the developer guide.
For information about the key-value pairs that AWS Glue consumes to set up your job, see the Special Parameters Used by AWS Glue topic in the developer guide.
- Description
-
- Type: string
Description of the job being defined.
- ExecutionProperty
-
- Type: ExecutionProperty structure
An
ExecutionPropertyspecifying the maximum number of concurrent runs allowed for this job. - GlueVersion
-
- Type: string
Glue version determines the versions of Apache Spark and Python that AWS Glue supports. The Python version indicates the version supported for jobs of type Spark.
For more information about the available AWS Glue versions and corresponding Spark and Python versions, see Glue version in the developer guide.
- LogUri
-
- Type: string
This field is reserved for future use.
- MaxCapacity
-
- Type: double
The number of AWS Glue data processing units (DPUs) that can be allocated when this job runs. A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB of memory. For more information, see the AWS Glue pricing page.
Do not set
Max Capacityif usingWorkerTypeandNumberOfWorkers.The value that can be allocated for
MaxCapacitydepends on whether you are running a Python shell job or an Apache Spark ETL job:-
When you specify a Python shell job (
JobCommand.Name="pythonshell"), you can allocate either 0.0625 or 1 DPU. The default is 0.0625 DPU. -
When you specify an Apache Spark ETL job (
JobCommand.Name="glueetl"), you can allocate from 2 to 100 DPUs. The default is 10 DPUs. This job type cannot have a fractional DPU allocation.
- MaxRetries
-
- Type: int
The maximum number of times to retry this job if it fails.
- NotificationProperty
-
- Type: NotificationProperty structure
Specifies the configuration properties of a job notification.
- NumberOfWorkers
-
- Type: int
The number of workers of a defined
workerTypethat are allocated when a job runs.The maximum number of workers you can define are 299 for
G.1X, and 149 forG.2X. - Role
-
- Type: string
The name or Amazon Resource Name (ARN) of the IAM role associated with this job (required).
- SecurityConfiguration
-
- Type: string
The name of the
SecurityConfigurationstructure to be used with this job. - Timeout
-
- Type: int
The job timeout in minutes. This is the maximum time that a job run can consume resources before it is terminated and enters
TIMEOUTstatus. The default is 2,880 minutes (48 hours). - WorkerType
-
- Type: string
The type of predefined worker that is allocated when a job runs. Accepts a value of Standard, G.1X, or G.2X.
-
For the
Standardworker type, each worker provides 4 vCPU, 16 GB of memory and a 50GB disk, and 2 executors per worker. -
For the
G.1Xworker type, each worker maps to 1 DPU (4 vCPU, 16 GB of memory, 64 GB disk), and provides 1 executor per worker. We recommend this worker type for memory-intensive jobs. -
For the
G.2Xworker type, each worker maps to 2 DPU (8 vCPU, 32 GB of memory, 128 GB disk), and provides 1 executor per worker. We recommend this worker type for memory-intensive jobs.
JsonClassifier
Description
A classifier for JSON content.
Members
- CreationTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The time that this classifier was registered.
- JsonPath
-
- Required: Yes
- Type: string
A
JsonPathstring defining the JSON data for the classifier to classify. AWS Glue supports a subset ofJsonPath, as described in Writing JsonPath Custom Classifiers. - LastUpdated
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The time that this classifier was last updated.
- Name
-
- Required: Yes
- Type: string
The name of the classifier.
- Version
-
- Type: long (int|float)
The version of this classifier.
LabelingSetGenerationTaskRunProperties
Description
Specifies configuration properties for a labeling set generation task run.
Members
LastCrawlInfo
Description
Status and error information about the most recent crawl.
Members
- ErrorMessage
-
- Type: string
If an error occurred, the error information about the last crawl.
- LogGroup
-
- Type: string
The log group for the last crawl.
- LogStream
-
- Type: string
The log stream for the last crawl.
- MessagePrefix
-
- Type: string
The prefix for a message about this crawl.
- StartTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The time at which the crawl started.
- Status
-
- Type: string
Status of the last crawl.
Location
Description
The location of resources.
Members
- DynamoDB
-
- Type: Array of CodeGenNodeArg structures
An Amazon DynamoDB table location.
- Jdbc
-
- Type: Array of CodeGenNodeArg structures
A JDBC location.
- S3
-
- Type: Array of CodeGenNodeArg structures
An Amazon Simple Storage Service (Amazon S3) location.
MLTransform
Description
A structure for a machine learning transform.
Members
- CreatedOn
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
A timestamp. The time and date that this machine learning transform was created.
- Description
-
- Type: string
A user-defined, long-form description text for the machine learning transform. Descriptions are not guaranteed to be unique and can be changed at any time.
- EvaluationMetrics
-
- Type: EvaluationMetrics structure
An
EvaluationMetricsobject. Evaluation metrics provide an estimate of the quality of your machine learning transform. - InputRecordTables
-
- Type: Array of GlueTable structures
A list of AWS Glue table definitions used by the transform.
- LabelCount
-
- Type: int
A count identifier for the labeling files generated by AWS Glue for this transform. As you create a better transform, you can iteratively download, label, and upload the labeling file.
- LastModifiedOn
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
A timestamp. The last point in time when this machine learning transform was modified.
- MaxCapacity
-
- Type: double
The number of AWS Glue data processing units (DPUs) that are allocated to task runs for this transform. You can allocate from 2 to 100 DPUs; the default is 10. A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB of memory. For more information, see the AWS Glue pricing page.
When the
WorkerTypefield is set to a value other thanStandard, theMaxCapacityfield is set automatically and becomes read-only. - MaxRetries
-
- Type: int
The maximum number of times to retry after an
MLTaskRunof the machine learning transform fails. - Name
-
- Type: string
A user-defined name for the machine learning transform. Names are not guaranteed unique and can be changed at any time.
- NumberOfWorkers
-
- Type: int
The number of workers of a defined
workerTypethat are allocated when a task of the transform runs. - Parameters
-
- Type: TransformParameters structure
A
TransformParametersobject. You can use parameters to tune (customize) the behavior of the machine learning transform by specifying what data it learns from and your preference on various tradeoffs (such as precious vs. recall, or accuracy vs. cost). - Role
-
- Type: string
The name or Amazon Resource Name (ARN) of the IAM role with the required permissions. This role needs permission to your Amazon Simple Storage Service (Amazon S3) sources, targets, temporary directory, scripts, and any libraries used by the task run for this transform.
- Schema
-
- Type: Array of SchemaColumn structures
A map of key-value pairs representing the columns and data types that this transform can run against. Has an upper bound of 100 columns.
- Status
-
- Type: string
The current status of the machine learning transform.
- Timeout
-
- Type: int
The timeout in minutes of the machine learning transform.
- TransformId
-
- Type: string
The unique transform ID that is generated for the machine learning transform. The ID is guaranteed to be unique and does not change.
- WorkerType
-
- Type: string
The type of predefined worker that is allocated when a task of this transform runs. Accepts a value of Standard, G.1X, or G.2X.
-
For the
Standardworker type, each worker provides 4 vCPU, 16 GB of memory and a 50GB disk, and 2 executors per worker. -
For the
G.1Xworker type, each worker provides 4 vCPU, 16 GB of memory and a 64GB disk, and 1 executor per worker. -
For the
G.2Xworker type, each worker provides 8 vCPU, 32 GB of memory and a 128GB disk, and 1 executor per worker.
MLTransformNotReadyException
Description
The machine learning transform is not ready to run.
Members
MappingEntry
Description
Defines a mapping.
Members
NoScheduleException
Description
There is no applicable schedule.
Members
Node
Description
A node represents an AWS Glue component like Trigger, Job etc. which is part of a workflow.
Members
- CrawlerDetails
-
- Type: CrawlerNodeDetails structure
Details of the crawler when the node represents a crawler.
- JobDetails
-
- Type: JobNodeDetails structure
Details of the Job when the node represents a Job.
- Name
-
- Type: string
The name of the AWS Glue component represented by the node.
- TriggerDetails
-
- Type: TriggerNodeDetails structure
Details of the Trigger when the node represents a Trigger.
- Type
-
- Type: string
The type of AWS Glue component represented by the node.
- UniqueId
-
- Type: string
The unique Id assigned to the node within the workflow.
NotificationProperty
Description
Specifies configuration properties of a notification.
Members
OperationTimeoutException
Order
Description
Specifies the sort order of a sorted column.
Members
Partition
Description
Represents a slice of table data.
Members
- CreationTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The time at which the partition was created.
- DatabaseName
-
- Type: string
The name of the catalog database in which to create the partition.
- LastAccessTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The last time at which the partition was accessed.
- LastAnalyzedTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The last time at which column statistics were computed for this partition.
- Parameters
-
- Type: Associative array of custom strings keys (KeyString) to strings
These key-value pairs define partition parameters.
- StorageDescriptor
-
- Type: StorageDescriptor structure
Provides information about the physical location where the partition is stored.
- TableName
-
- Type: string
The name of the database table in which to create the partition.
- Values
-
- Type: Array of strings
The values of the partition.
PartitionError
Description
Contains information about a partition error.
Members
- ErrorDetail
-
- Type: ErrorDetail structure
The details about the partition error.
- PartitionValues
-
- Type: Array of strings
The values that define the partition.
PartitionInput
Description
The structure used to create and update a partition.
Members
- LastAccessTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The last time at which the partition was accessed.
- LastAnalyzedTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The last time at which column statistics were computed for this partition.
- Parameters
-
- Type: Associative array of custom strings keys (KeyString) to strings
These key-value pairs define partition parameters.
- StorageDescriptor
-
- Type: StorageDescriptor structure
Provides information about the physical location where the partition is stored.
- Values
-
- Type: Array of strings
The values of the partition. Although this parameter is not required by the SDK, you must specify this parameter for a valid input.
PartitionValueList
Description
Contains a list of values defining partitions.
Members
PhysicalConnectionRequirements
Description
Specifies the physical requirements for a connection.
Members
- AvailabilityZone
-
- Type: string
The connection's Availability Zone. This field is redundant because the specified subnet implies the Availability Zone to be used. Currently the field must be populated, but it will be deprecated in the future.
- SecurityGroupIdList
-
- Type: Array of strings
The security group ID list used by the connection.
- SubnetId
-
- Type: string
The subnet ID used by the connection.
Predecessor
Description
A job run that was used in the predicate of a conditional trigger that triggered this job run.
Members
Predicate
Description
Defines the predicate of the trigger, which determines when it fires.
Members
- Conditions
-
- Type: Array of Condition structures
A list of the conditions that determine when the trigger will fire.
- Logical
-
- Type: string
An optional field if only one condition is listed. If multiple conditions are listed, then this field is required.
PrincipalPermissions
Description
Permissions granted to a principal.
Members
- Permissions
-
- Type: Array of strings
The permissions that are granted to the principal.
- Principal
-
- Type: DataLakePrincipal structure
The principal who is granted permissions.
PropertyPredicate
Description
Defines a property predicate.
Members
ResourceNumberLimitExceededException
Description
A resource numerical limit was exceeded.
Members
ResourceUri
Description
The URIs for function resources.
Members
S3Encryption
Description
Specifies how Amazon Simple Storage Service (Amazon S3) data should be encrypted.
Members
S3Target
Description
Specifies a data store in Amazon Simple Storage Service (Amazon S3).
Members
- Exclusions
-
- Type: Array of strings
A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler.
- Path
-
- Type: string
The path to the Amazon S3 target.
Schedule
Description
A scheduling object using a cron statement to schedule an event.
Members
- ScheduleExpression
-
- Type: string
A
cronexpression used to specify the schedule. For more information, see Time-Based Schedules for Jobs and Crawlers. For example, to run something every day at 12:15 UTC, specifycron(15 12 * * ? *). - State
-
- Type: string
The state of the schedule.
SchedulerNotRunningException
Description
The specified scheduler is not running.
Members
SchedulerRunningException
Description
The specified scheduler is already running.
Members
SchedulerTransitioningException
Description
The specified scheduler is transitioning.
Members
SchemaChangePolicy
Description
A policy that specifies update and deletion behaviors for the crawler.
Members
SchemaColumn
Description
A key-value pair representing a column and data type that this transform can run against. The Schema parameter of the MLTransform may contain up to 100 of these structures.
Members
SecurityConfiguration
Description
Specifies a security configuration.
Members
- CreatedTimeStamp
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The time at which this security configuration was created.
- EncryptionConfiguration
-
- Type: EncryptionConfiguration structure
The encryption configuration associated with this security configuration.
- Name
-
- Type: string
The name of the security configuration.
Segment
Description
Defines a non-overlapping region of a table's partitions, allowing multiple requests to be executed in parallel.
Members
SerDeInfo
Description
Information about a serialization/deserialization program (SerDe) that serves as an extractor and loader.
Members
- Name
-
- Type: string
Name of the SerDe.
- Parameters
-
- Type: Associative array of custom strings keys (KeyString) to strings
These key-value pairs define initialization parameters for the SerDe.
- SerializationLibrary
-
- Type: string
Usually the class that implements the SerDe. An example is
org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe.
SkewedInfo
Description
Specifies skewed values in a table. Skewed values are those that occur with very high frequency.
Members
- SkewedColumnNames
-
- Type: Array of strings
A list of names of columns that contain skewed values.
- SkewedColumnValueLocationMaps
-
- Type: Associative array of custom strings keys (ColumnValuesString) to strings
A mapping of skewed values to the columns that contain them.
- SkewedColumnValues
-
- Type: Array of strings
A list of values that appear so frequently as to be considered skewed.
SortCriterion
StorageDescriptor
Description
Describes the physical storage of table data.
Members
- BucketColumns
-
- Type: Array of strings
A list of reducer grouping columns, clustering columns, and bucketing columns in the table.
- Columns
-
- Type: Array of Column structures
A list of the
Columnsin the table. - Compressed
-
- Type: boolean
Trueif the data in the table is compressed, orFalseif not. - InputFormat
-
- Type: string
The input format:
SequenceFileInputFormat(binary), orTextInputFormat, or a custom format. - Location
-
- Type: string
The physical location of the table. By default, this takes the form of the warehouse location, followed by the database location in the warehouse, followed by the table name.
- NumberOfBuckets
-
- Type: int
Must be specified if the table contains any dimension columns.
- OutputFormat
-
- Type: string
The output format:
SequenceFileOutputFormat(binary), orIgnoreKeyTextOutputFormat, or a custom format. - Parameters
-
- Type: Associative array of custom strings keys (KeyString) to strings
The user-supplied properties in key-value form.
- SerdeInfo
-
- Type: SerDeInfo structure
The serialization/deserialization (SerDe) information.
- SkewedInfo
-
- Type: SkewedInfo structure
The information about values that appear frequently in a column (skewed values).
- SortColumns
-
- Type: Array of Order structures
A list specifying the sort order of each bucket in the table.
- StoredAsSubDirectories
-
- Type: boolean
Trueif the table data is stored in subdirectories, orFalseif not.
Table
Description
Represents a collection of related data organized in columns and rows.
Members
- CreateTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The time when the table definition was created in the Data Catalog.
- CreatedBy
-
- Type: string
The person or entity who created the table.
- DatabaseName
-
- Type: string
The name of the database where the table metadata resides. For Hive compatibility, this must be all lowercase.
- Description
-
- Type: string
A description of the table.
- IsRegisteredWithLakeFormation
-
- Type: boolean
Indicates whether the table has been registered with AWS Lake Formation.
- LastAccessTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The last time that the table was accessed. This is usually taken from HDFS, and might not be reliable.
- LastAnalyzedTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The last time that column statistics were computed for this table.
- Name
-
- Required: Yes
- Type: string
The table name. For Hive compatibility, this must be entirely lowercase.
- Owner
-
- Type: string
The owner of the table.
- Parameters
-
- Type: Associative array of custom strings keys (KeyString) to strings
These key-value pairs define properties associated with the table.
- PartitionKeys
-
- Type: Array of Column structures
A list of columns by which the table is partitioned. Only primitive types are supported as partition keys.
When you create a table used by Amazon Athena, and you do not specify any
partitionKeys, you must at least set the value ofpartitionKeysto an empty list. For example:"PartitionKeys": [] - Retention
-
- Type: int
The retention time for this table.
- StorageDescriptor
-
- Type: StorageDescriptor structure
A storage descriptor containing information about the physical storage of this table.
- TableType
-
- Type: string
The type of this table (
EXTERNAL_TABLE,VIRTUAL_VIEW, etc.). - UpdateTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The last time that the table was updated.
- ViewExpandedText
-
- Type: string
If the table is a view, the expanded text of the view; otherwise
null. - ViewOriginalText
-
- Type: string
If the table is a view, the original text of the view; otherwise
null.
TableError
Description
An error record for table operations.
Members
- ErrorDetail
-
- Type: ErrorDetail structure
The details about the error.
- TableName
-
- Type: string
The name of the table. For Hive compatibility, this must be entirely lowercase.
TableInput
Description
A structure used to define a table.
Members
- Description
-
- Type: string
A description of the table.
- LastAccessTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The last time that the table was accessed.
- LastAnalyzedTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The last time that column statistics were computed for this table.
- Name
-
- Required: Yes
- Type: string
The table name. For Hive compatibility, this is folded to lowercase when it is stored.
- Owner
-
- Type: string
The table owner.
- Parameters
-
- Type: Associative array of custom strings keys (KeyString) to strings
These key-value pairs define properties associated with the table.
- PartitionKeys
-
- Type: Array of Column structures
A list of columns by which the table is partitioned. Only primitive types are supported as partition keys.
When you create a table used by Amazon Athena, and you do not specify any
partitionKeys, you must at least set the value ofpartitionKeysto an empty list. For example:"PartitionKeys": [] - Retention
-
- Type: int
The retention time for this table.
- StorageDescriptor
-
- Type: StorageDescriptor structure
A storage descriptor containing information about the physical storage of this table.
- TableType
-
- Type: string
The type of this table (
EXTERNAL_TABLE,VIRTUAL_VIEW, etc.). - ViewExpandedText
-
- Type: string
If the table is a view, the expanded text of the view; otherwise
null. - ViewOriginalText
-
- Type: string
If the table is a view, the original text of the view; otherwise
null.
TableVersion
Description
Specifies a version of a table.
Members
- Table
-
- Type: Table structure
The table in question.
- VersionId
-
- Type: string
The ID value that identifies this table version. A
VersionIdis a string representation of an integer. Each version is incremented by 1.
TableVersionError
Description
An error record for table-version operations.
Members
- ErrorDetail
-
- Type: ErrorDetail structure
The details about the error.
- TableName
-
- Type: string
The name of the table in question.
- VersionId
-
- Type: string
The ID value of the version in question. A
VersionIDis a string representation of an integer. Each version is incremented by 1.
TaskRun
Description
The sampling parameters that are associated with the machine learning transform.
Members
- CompletedOn
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The last point in time that the requested task run was completed.
- ErrorString
-
- Type: string
The list of error strings associated with this task run.
- ExecutionTime
-
- Type: int
The amount of time (in seconds) that the task run consumed resources.
- LastModifiedOn
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The last point in time that the requested task run was updated.
- LogGroupName
-
- Type: string
The names of the log group for secure logging, associated with this task run.
- Properties
-
- Type: TaskRunProperties structure
Specifies configuration properties associated with this task run.
- StartedOn
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The date and time that this task run started.
- Status
-
- Type: string
The current status of the requested task run.
- TaskRunId
-
- Type: string
The unique identifier for this task run.
- TransformId
-
- Type: string
The unique identifier for the transform.
TaskRunFilterCriteria
Description
The criteria that are used to filter the task runs for the machine learning transform.
Members
- StartedAfter
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
Filter on task runs started after this date.
- StartedBefore
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
Filter on task runs started before this date.
- Status
-
- Type: string
The current status of the task run.
- TaskRunType
-
- Type: string
The type of task run.
TaskRunProperties
Description
The configuration properties for the task run.
Members
- ExportLabelsTaskRunProperties
-
- Type: ExportLabelsTaskRunProperties structure
The configuration properties for an exporting labels task run.
- FindMatchesTaskRunProperties
-
- Type: FindMatchesTaskRunProperties structure
The configuration properties for a find matches task run.
- ImportLabelsTaskRunProperties
-
- Type: ImportLabelsTaskRunProperties structure
The configuration properties for an importing labels task run.
- LabelingSetGenerationTaskRunProperties
-
- Type: LabelingSetGenerationTaskRunProperties structure
The configuration properties for a labeling set generation task run.
- TaskType
-
- Type: string
The type of task run.
TaskRunSortCriteria
Description
The sorting criteria that are used to sort the list of task runs for the machine learning transform.
Members
TransformFilterCriteria
Description
The criteria used to filter the machine learning transforms.
Members
- CreatedAfter
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The time and date after which the transforms were created.
- CreatedBefore
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The time and date before which the transforms were created.
- LastModifiedAfter
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
Filter on transforms last modified after this date.
- LastModifiedBefore
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
Filter on transforms last modified before this date.
- Name
-
- Type: string
A unique transform name that is used to filter the machine learning transforms.
- Schema
-
- Type: Array of SchemaColumn structures
Filters on datasets with a specific schema. The
Map<Column, Type>object is an array of key-value pairs representing the schema this transform accepts, whereColumnis the name of a column, andTypeis the type of the data such as an integer or string. Has an upper bound of 100 columns. - Status
-
- Type: string
Filters the list of machine learning transforms by the last known status of the transforms (to indicate whether a transform can be used or not). One of "NOT_READY", "READY", or "DELETING".
- TransformType
-
- Type: string
The type of machine learning transform that is used to filter the machine learning transforms.
TransformParameters
Description
The algorithm-specific parameters that are associated with the machine learning transform.
Members
- FindMatchesParameters
-
- Type: FindMatchesParameters structure
The parameters for the find matches algorithm.
- TransformType
-
- Required: Yes
- Type: string
The type of machine learning transform.
For information about the types of machine learning transforms, see Creating Machine Learning Transforms.
TransformSortCriteria
Description
The sorting criteria that are associated with the machine learning transform.
Members
Trigger
Description
Information about a specific trigger.
Members
- Actions
-
- Type: Array of Action structures
The actions initiated by this trigger.
- Description
-
- Type: string
A description of this trigger.
- Id
-
- Type: string
Reserved for future use.
- Name
-
- Type: string
The name of the trigger.
- Predicate
-
- Type: Predicate structure
The predicate of this trigger, which defines when it will fire.
- Schedule
-
- Type: string
A
cronexpression used to specify the schedule (see Time-Based Schedules for Jobs and Crawlers. For example, to run something every day at 12:15 UTC, you would specify:cron(15 12 * * ? *). - State
-
- Type: string
The current state of the trigger.
- Type
-
- Type: string
The type of trigger that this is.
- WorkflowName
-
- Type: string
The name of the workflow associated with the trigger.
TriggerNodeDetails
Description
The details of a Trigger node present in the workflow.
Members
- Trigger
-
- Type: Trigger structure
The information of the trigger represented by the trigger node.
TriggerUpdate
Description
A structure used to provide information used to update a trigger. This object updates the previous trigger definition by overwriting it completely.
Members
- Actions
-
- Type: Array of Action structures
The actions initiated by this trigger.
- Description
-
- Type: string
A description of this trigger.
- Name
-
- Type: string
Reserved for future use.
- Predicate
-
- Type: Predicate structure
The predicate of this trigger, which defines when it will fire.
- Schedule
-
- Type: string
A
cronexpression used to specify the schedule (see Time-Based Schedules for Jobs and Crawlers. For example, to run something every day at 12:15 UTC, you would specify:cron(15 12 * * ? *).
UpdateCsvClassifierRequest
Description
Specifies a custom CSV classifier to be updated.
Members
- AllowSingleColumn
-
- Type: boolean
Enables the processing of files that contain only one column.
- ContainsHeader
-
- Type: string
Indicates whether the CSV file contains a header.
- Delimiter
-
- Type: string
A custom symbol to denote what separates each column entry in the row.
- DisableValueTrimming
-
- Type: boolean
Specifies not to trim values before identifying the type of column values. The default value is true.
- Header
-
- Type: Array of strings
A list of strings representing column names.
- Name
-
- Required: Yes
- Type: string
The name of the classifier.
- QuoteSymbol
-
- Type: string
A custom symbol to denote what combines content into a single column value. It must be different from the column delimiter.
UpdateGrokClassifierRequest
Description
Specifies a grok classifier to update when passed to UpdateClassifier.
Members
- Classification
-
- Type: string
An identifier of the data format that the classifier matches, such as Twitter, JSON, Omniture logs, Amazon CloudWatch Logs, and so on.
- CustomPatterns
-
- Type: string
Optional custom grok patterns used by this classifier.
- GrokPattern
-
- Type: string
The grok pattern used by this classifier.
- Name
-
- Required: Yes
- Type: string
The name of the
GrokClassifier.
UpdateJsonClassifierRequest
Description
Specifies a JSON classifier to be updated.
Members
- JsonPath
-
- Type: string
A
JsonPathstring defining the JSON data for the classifier to classify. AWS Glue supports a subset ofJsonPath, as described in Writing JsonPath Custom Classifiers. - Name
-
- Required: Yes
- Type: string
The name of the classifier.
UpdateXMLClassifierRequest
Description
Specifies an XML classifier to be updated.
Members
- Classification
-
- Type: string
An identifier of the data format that the classifier matches.
- Name
-
- Required: Yes
- Type: string
The name of the classifier.
- RowTag
-
- Type: string
The XML tag designating the element that contains each record in an XML document being parsed. This cannot identify a self-closing element (closed by
/>). An empty row element that contains only attributes can be parsed as long as it ends with a closing tag (for example,<row item_a="A" item_b="B"></row>is okay, but<row item_a="A" item_b="B" />is not).
UserDefinedFunction
Description
Represents the equivalent of a Hive user-defined function (UDF) definition.
Members
- ClassName
-
- Type: string
The Java class that contains the function code.
- CreateTime
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The time at which the function was created.
- FunctionName
-
- Type: string
The name of the function.
- OwnerName
-
- Type: string
The owner of the function.
- OwnerType
-
- Type: string
The owner type.
- ResourceUris
-
- Type: Array of ResourceUri structures
The resource URIs for the function.
UserDefinedFunctionInput
Description
A structure used to create or update a user-defined function.
Members
- ClassName
-
- Type: string
The Java class that contains the function code.
- FunctionName
-
- Type: string
The name of the function.
- OwnerName
-
- Type: string
The owner of the function.
- OwnerType
-
- Type: string
The owner type.
- ResourceUris
-
- Type: Array of ResourceUri structures
The resource URIs for the function.
ValidationException
Description
A value could not be validated.
Members
VersionMismatchException
Description
There was a version conflict.
Members
Workflow
Description
A workflow represents a flow in which AWS Glue components should be executed to complete a logical task.
Members
- CreatedOn
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The date and time when the workflow was created.
- DefaultRunProperties
-
- Type: Associative array of custom strings keys (IdString) to strings
A collection of properties to be used as part of each execution of the workflow.
- Description
-
- Type: string
A description of the workflow.
- Graph
-
- Type: WorkflowGraph structure
The graph representing all the AWS Glue components that belong to the workflow as nodes and directed connections between them as edges.
- LastModifiedOn
-
- Type: timestamp (string|DateTime or anything parsable by strtotime)
The date and time when the workflow was last modified.
- LastRun
-
- Type: WorkflowRun structure
The information about the last execution of the workflow.
- Name
-
- Type: string
The name of the workflow representing the flow.
WorkflowGraph
Description
A workflow graph represents