AWS Glue API

This section describes data types and primitives used by AWS Glue SDKs and Tools. There are three general ways to interact with AWS Glue programmatically outside of the AWS Management Console, each with its own documentation:

• Language SDK libraries allow you to access AWS resources from common programming languages. Find more information at Tools to Build on AWS.

• The AWS CLI allows you to access AWS resources from the command line. Find more information at AWS CLI Command Reference.

• AWS CloudFormation allows you to define a set of AWS resources to be provisioned together consistently. Find more information at AWS CloudFormation: AWS Glue resource type reference.

This section documents shared primitives independently of these SDKs and Tools. Tools use the AWS Glue Web API Reference to communicate with AWS.

Contents

• Security APIs in AWS Glue
  • Data types
  • DataCatalogEncryptionSettings structure
  • EncryptionAtRest structure
  • ConnectionPasswordEncryption structure
  • EncryptionConfiguration structure
  • S3Encryption structure
  • CloudWatchEncryption structure
  • JobBookmarksEncryption structure
  • SecurityConfiguration structure
  • GluePolicy structure
  • Operations
  • GetDataCatalogEncryptionSettings action (Python: get_data_catalog_encryption_settings)
  • PutDataCatalogEncryptionSettings action (Python: put_data_catalog_encryption_settings)
  • PutResourcePolicy action (Python: put_resource_policy)
  • GetResourcePolicy action (Python: get_resource_policy)
  • DeleteResourcePolicy action (Python: delete_resource_policy)
  • CreateSecurityConfiguration action (Python: create_security_configuration)
  • DeleteSecurityConfiguration action (Python: delete_security_configuration)
  • GetSecurityConfiguration action (Python: get_security_configuration)
  • GetSecurityConfigurations action (Python: get_security_configurations)
  • GetResourcePolicies action (Python: get_resource_policies)
• Catalog API
  • Database API
    • Data types
    • Database structure
    • DatabaseInput structure
    • PrincipalPermissions structure
    • DataLakePrincipal structure
    • DatabaseIdentifier structure
    • Operations
    • CreateDatabase action (Python: create_database)
    • UpdateDatabase action (Python: update_database)
    • DeleteDatabase action (Python: delete_database)
    • GetDatabase action (Python: get_database)
    • GetDatabases action (Python: get_databases)
  • Table API
    • Data types
    • Table structure
    • TableInput structure
    • Column structure
    • StorageDescriptor structure
    • SchemaReference structure
    • SerDeInfo structure
    • Order structure
    • SkewedInfo structure
    • TableVersion structure
    • TableError structure
    • TableVersionError structure
    • SortCriterion structure
    • TableIdentifier structure
    • KeySchemaElement structure
    • PartitionIndex structure
    • PartitionIndexDescriptor structure
    • BackfillError structure
    • Operations
    • CreateTable action (Python: create_table)
    • UpdateTable action (Python: update_table)
    • DeleteTable action (Python: delete_table)
    • BatchDeleteTable action (Python: batch_delete_table)
    • GetTable action (Python: get_table)
    • GetTables action (Python: get_tables)
    • GetTableVersion action (Python: get_table_version)
    • GetTableVersions action (Python: get_table_versions)
    • DeleteTableVersion action (Python: delete_table_version)
    • BatchDeleteTableVersion action (Python: batch_delete_table_version)
    • SearchTables action (Python: search_tables)
    • GetPartitionIndexes action (Python: get_partition_indexes)
    • CreatePartitionIndex action (Python: create_partition_index)
    • DeletePartitionIndex action (Python: delete_partition_index)
    • GetColumnStatisticsForTable action (Python: get_column_statistics_for_table)
    • UpdateColumnStatisticsForTable action (Python: update_column_statistics_for_table)
    • DeleteColumnStatisticsForTable action (Python: delete_column_statistics_for_table)
  • Partition API
    • Data types
    • Partition structure
    • PartitionInput structure
    • PartitionSpecWithSharedStorageDescriptor structure
    • PartitionListComposingSpec structure
    • PartitionSpecProxy structure
    • PartitionValueList structure
    • Segment structure
    • PartitionError structure
    • BatchUpdatePartitionFailureEntry structure
    • BatchUpdatePartitionRequestEntry structure
    • Operations
    • CreatePartition action (Python: create_partition)
    • BatchCreatePartition action (Python: batch_create_partition)
    • UpdatePartition action (Python: update_partition)
    • DeletePartition action (Python: delete_partition)
    • BatchDeletePartition action (Python: batch_delete_partition)
    • GetPartition action (Python: get_partition)
    • GetPartitions action (Python: get_partitions)
    • BatchGetPartition action (Python: batch_get_partition)
    • BatchUpdatePartition action (Python: batch_update_partition)
    • GetColumnStatisticsForPartition action (Python: get_column_statistics_for_partition)
    • UpdateColumnStatisticsForPartition action (Python: update_column_statistics_for_partition)
    • DeleteColumnStatisticsForPartition action (Python: delete_column_statistics_for_partition)
  • Connection API
    • Data types
    • Connection structure
    • ConnectionInput structure
    • PhysicalConnectionRequirements structure
    • GetConnectionsFilter structure
    • Operations
    • CreateConnection action (Python: create_connection)
    • DeleteConnection action (Python: delete_connection)
    • GetConnection action (Python: get_connection)
    • GetConnections action (Python: get_connections)
    • UpdateConnection action (Python: update_connection)
    • BatchDeleteConnection action (Python: batch_delete_connection)
  • User-defined Function API
    • Data types
    • UserDefinedFunction structure
    • UserDefinedFunctionInput structure
    • Operations
    • CreateUserDefinedFunction action (Python: create_user_defined_function)
    • UpdateUserDefinedFunction action (Python: update_user_defined_function)
    • DeleteUserDefinedFunction action (Python: delete_user_defined_function)
    • GetUserDefinedFunction action (Python: get_user_defined_function)
    • GetUserDefinedFunctions action (Python: get_user_defined_functions)
  • Importing an Athena catalog to AWS Glue
    • Data types
    • CatalogImportStatus structure
    • Operations
    • ImportCatalogToGlue action (Python: import_catalog_to_glue)
    • GetCatalogImportStatus action (Python: get_catalog_import_status)
• Crawlers and classifiers API
  • Classifier API
    • Data types
    • Classifier structure
    • GrokClassifier structure
    • XMLClassifier structure
    • JsonClassifier structure
    • CsvClassifier structure
    • CreateGrokClassifierRequest structure
    • UpdateGrokClassifierRequest structure
    • CreateXMLClassifierRequest structure
    • UpdateXMLClassifierRequest structure
    • CreateJsonClassifierRequest structure
    • UpdateJsonClassifierRequest structure
    • CreateCsvClassifierRequest structure
    • UpdateCsvClassifierRequest structure
    • Operations
    • CreateClassifier action (Python: create_classifier)
    • DeleteClassifier action (Python: delete_classifier)
    • GetClassifier action (Python: get_classifier)
    • GetClassifiers action (Python: get_classifiers)
    • UpdateClassifier action (Python: update_classifier)
  • Crawler API
    • Data types
    • Crawler structure
    • Schedule structure
    • CrawlerTargets structure
    • S3Target structure
    • JdbcTarget structure
    • MongoDBTarget structure
    • DynamoDBTarget structure
    • DeltaTarget structure
    • CatalogTarget structure
    • CrawlerMetrics structure
    • CrawlerHistory structure
    • CrawlsFilter structure
    • SchemaChangePolicy structure
    • LastCrawlInfo structure
    • RecrawlPolicy structure
    • LineageConfiguration structure
    • LakeFormationConfiguration structure
    • Operations
    • CreateCrawler action (Python: create_crawler)
    • DeleteCrawler action (Python: delete_crawler)
    • GetCrawler action (Python: get_crawler)
    • GetCrawlers action (Python: get_crawlers)
    • GetCrawlerMetrics action (Python: get_crawler_metrics)
    • UpdateCrawler action (Python: update_crawler)
    • StartCrawler action (Python: start_crawler)
    • StopCrawler action (Python: stop_crawler)
    • BatchGetCrawlers action (Python: batch_get_crawlers)
    • ListCrawlers action (Python: list_crawlers)
    • ListCrawls action (Python: list_crawls)
  • Crawler scheduler API
    • Data types
    • Schedule structure
    • Operations
    • UpdateCrawlerSchedule action (Python: update_crawler_schedule)
    • StartCrawlerSchedule action (Python: start_crawler_schedule)
    • StopCrawlerSchedule action (Python: stop_crawler_schedule)
• Autogenerating ETL Scripts API
  • Data types
  • CodeGenNode structure
  • CodeGenNodeArg structure
  • CodeGenEdge structure
  • Location structure
  • CatalogEntry structure
  • MappingEntry structure
  • Operations
  • CreateScript action (Python: create_script)
  • GetDataflowGraph action (Python: get_dataflow_graph)
  • GetMapping action (Python: get_mapping)
  • GetPlan action (Python: get_plan)
• Visual job API
  • Data types
  • CodeGenConfigurationNode structure
  • JDBCConnectorOptions structure
  • StreamingDataPreviewOptions structure
  • AthenaConnectorSource structure
  • JDBCConnectorSource structure
  • SparkConnectorSource structure
  • CatalogSource structure
  • MySQLCatalogSource structure
  • PostgreSQLCatalogSource structure
  • OracleSQLCatalogSource structure
  • MicrosoftSQLServerCatalogSource structure
  • CatalogKinesisSource structure
  • DirectKinesisSource structure
  • KinesisStreamingSourceOptions structure
  • CatalogKafkaSource structure
  • DirectKafkaSource structure
  • KafkaStreamingSourceOptions structure
  • RedshiftSource structure
  • S3CatalogSource structure
  • GovernedCatalogSource structure
  • S3SourceAdditionalOptions structure
  • S3CsvSource structure
  • S3DirectSourceAdditionalOptions structure
  • S3JsonSource structure
  • S3ParquetSource structure
  • DynamoDBCatalogSource structure
  • RelationalCatalogSource structure
  • S3CatalogHudiSource structure
  • CatalogHudiSource structure
  • S3HudiSource structure
  • JDBCConnectorTarget structure
  • SparkConnectorTarget structure
  • BasicCatalogTarget structure
  • MySQLCatalogTarget structure
  • PostgreSQLCatalogTarget structure
  • OracleSQLCatalogTarget structure
  • MicrosoftSQLServerCatalogTarget structure
  • RedshiftTarget structure
  • UpsertRedshiftTargetOptions structure
  • S3CatalogTarget structure
  • GovernedCatalogTarget structure
  • S3GlueParquetTarget structure
  • CatalogSchemaChangePolicy structure
  • S3DirectTarget structure
  • S3HudiCatalogTarget structure
  • S3HudiDirectTarget structure
  • DirectSchemaChangePolicy structure
  • ApplyMapping structure
  • Mapping structure
  • SelectFields structure
  • DropFields structure
  • RenameField structure
  • Spigot structure
  • Join structure
  • JoinColumn structure
  • SplitFields structure
  • SelectFromCollection structure
  • FillMissingValues structure
  • Filter structure
  • FilterExpression structure
  • FilterValue structure
  • CustomCode structure
  • SparkSQL structure
  • SqlAlias structure
  • DropNullFields structure
  • NullCheckBoxList structure
  • NullValueField structure
  • Datatype structure
  • Merge structure
  • DropDuplicates structure
  • Union structure
  • PIIDetection structure
  • Aggregate structure
  • AggregateOperation structure
  • GlueSchema structure
  • GlueStudioSchemaColumn structure
  • GlueStudioColumn structure
  • DynamicTransform structure
  • TransformConfigParameter structure
  • EvaluateDataQuality structure
  • DQResultsPublishingOptions structure
  • DQStopJobOnFailureOptions structure
• Jobs API
  • Jobs
    • Data types
    • Job structure
    • ExecutionProperty structure
    • NotificationProperty structure
    • JobCommand structure
    • ConnectionsList structure
    • JobUpdate structure
    • SourceControlDetails structure
    • Operations
    • CreateJob action (Python: create_job)
    • UpdateJob action (Python: update_job)
    • GetJob action (Python: get_job)
    • GetJobs action (Python: get_jobs)
    • DeleteJob action (Python: delete_job)
    • ListJobs action (Python: list_jobs)
    • BatchGetJobs action (Python: batch_get_jobs)
    • UpdateSourceControlFromJob action (Python: update_source_control_from_job)
    • UpdateJobFromSourceControl action (Python: update_job_from_source_control)
  • Job runs
    • Data types
    • JobRun structure
    • Predecessor structure
    • JobBookmarkEntry structure
    • BatchStopJobRunSuccessfulSubmission structure
    • BatchStopJobRunError structure
    • Operations
    • StartJobRun action (Python: start_job_run)
    • BatchStopJobRun action (Python: batch_stop_job_run)
    • GetJobRun action (Python: get_job_run)
    • GetJobRuns action (Python: get_job_runs)
    • GetJobBookmark action (Python: get_job_bookmark)
    • GetJobBookmarks action (Python: get_job_bookmarks)
    • ResetJobBookmark action (Python: reset_job_bookmark)
  • Triggers
    • Data types
    • Trigger structure
    • TriggerUpdate structure
    • Predicate structure
    • Condition structure
    • Action structure
    • EventBatchingCondition structure
    • Operations
    • CreateTrigger action (Python: create_trigger)
    • StartTrigger action (Python: start_trigger)
    • GetTrigger action (Python: get_trigger)
    • GetTriggers action (Python: get_triggers)
    • UpdateTrigger action (Python: update_trigger)
    • StopTrigger action (Python: stop_trigger)
    • DeleteTrigger action (Python: delete_trigger)
    • ListTriggers action (Python: list_triggers)
    • BatchGetTriggers action (Python: batch_get_triggers)
• Interactive sessions API
  • Data types
  • Session structure
  • SessionCommand structure
  • Statement structure
  • StatementOutput structure
  • StatementOutputData structure
  • Operations
  • CreateSession action (Python: create_session)
  • StopSession action (Python: stop_session)
  • DeleteSession action (Python: delete_session)
  • GetSession action (Python: get_session)
  • ListSessions action (Python: list_sessions)
  • RunStatement action (Python: run_statement)
  • CancelStatement action (Python: cancel_statement)
  • GetStatement action (Python: get_statement)
  • ListStatements action (Python: list_statements)
• Development endpoints API
  • Data types
  • DevEndpoint structure
  • DevEndpointCustomLibraries structure
  • Operations
  • CreateDevEndpoint action (Python: create_dev_endpoint)
  • UpdateDevEndpoint action (Python: update_dev_endpoint)
  • DeleteDevEndpoint action (Python: delete_dev_endpoint)
  • GetDevEndpoint action (Python: get_dev_endpoint)
  • GetDevEndpoints action (Python: get_dev_endpoints)
  • BatchGetDevEndpoints action (Python: batch_get_dev_endpoints)
  • ListDevEndpoints action (Python: list_dev_endpoints)
• Schema registry
  • Data types
  • RegistryId structure
  • RegistryListItem structure
  • MetadataInfo structure
  • OtherMetadataValueListItem structure
  • SchemaListItem structure
  • SchemaVersionListItem structure
  • MetadataKeyValuePair structure
  • SchemaVersionErrorItem structure
  • ErrorDetails structure
  • SchemaVersionNumber structure
  • SchemaId structure
  • Operations
  • CreateRegistry action (Python: create_registry)
  • CreateSchema action (Python: create_schema)
  • GetSchema action (Python: get_schema)
  • ListSchemaVersions action (Python: list_schema_versions)
  • GetSchemaVersion action (Python: get_schema_version)
  • GetSchemaVersionsDiff action (Python: get_schema_versions_diff)
  • ListRegistries action (Python: list_registries)
  • ListSchemas action (Python: list_schemas)
  • RegisterSchemaVersion action (Python: register_schema_version)
  • UpdateSchema action (Python: update_schema)
  • CheckSchemaVersionValidity action (Python: check_schema_version_validity)
  • UpdateRegistry action (Python: update_registry)
  • GetSchemaByDefinition action (Python: get_schema_by_definition)
  • GetRegistry action (Python: get_registry)
  • PutSchemaVersionMetadata action (Python: put_schema_version_metadata)
  • QuerySchemaVersionMetadata action (Python: query_schema_version_metadata)
  • RemoveSchemaVersionMetadata action (Python: remove_schema_version_metadata)
  • DeleteRegistry action (Python: delete_registry)
  • DeleteSchema action (Python: delete_schema)
  • DeleteSchemaVersions action (Python: delete_schema_versions)
• Workflows
  • Data types
  • JobNodeDetails structure
  • CrawlerNodeDetails structure
  • TriggerNodeDetails structure
  • Crawl structure
  • Node structure
  • Edge structure
  • Workflow structure
  • WorkflowGraph structure
  • WorkflowRun structure
  • WorkflowRunStatistics structure
  • StartingEventBatchCondition structure
  • Blueprint structure
  • BlueprintDetails structure
  • LastActiveDefinition structure
  • BlueprintRun structure
  • Operations
  • CreateWorkflow action (Python: create_workflow)
  • UpdateWorkflow action (Python: update_workflow)
  • DeleteWorkflow action (Python: delete_workflow)
  • GetWorkflow action (Python: get_workflow)
  • ListWorkflows action (Python: list_workflows)
  • BatchGetWorkflows action (Python: batch_get_workflows)
  • GetWorkflowRun action (Python: get_workflow_run)
  • GetWorkflowRuns action (Python: get_workflow_runs)
  • GetWorkflowRunProperties action (Python: get_workflow_run_properties)
  • PutWorkflowRunProperties action (Python: put_workflow_run_properties)
  • CreateBlueprint action (Python: create_blueprint)
  • UpdateBlueprint action (Python: update_blueprint)
  • DeleteBlueprint action (Python: delete_blueprint)
  • ListBlueprints action (Python: list_blueprints)
  • BatchGetBlueprints action (Python: batch_get_blueprints)
  • StartBlueprintRun action (Python: start_blueprint_run)
  • GetBlueprintRun action (Python: get_blueprint_run)
  • GetBlueprintRuns action (Python: get_blueprint_runs)
  • StartWorkflowRun action (Python: start_workflow_run)
  • StopWorkflowRun action (Python: stop_workflow_run)
  • ResumeWorkflowRun action (Python: resume_workflow_run)
• Machine learning API
  • Data types
  • TransformParameters structure
  • EvaluationMetrics structure
  • MLTransform structure
  • FindMatchesParameters structure
  • FindMatchesMetrics structure
  • ConfusionMatrix structure
  • GlueTable structure
  • TaskRun structure
  • TransformFilterCriteria structure
  • TransformSortCriteria structure
  • TaskRunFilterCriteria structure
  • TaskRunSortCriteria structure
  • TaskRunProperties structure
  • FindMatchesTaskRunProperties structure
  • ImportLabelsTaskRunProperties structure
  • ExportLabelsTaskRunProperties structure
  • LabelingSetGenerationTaskRunProperties structure
  • SchemaColumn structure
  • TransformEncryption structure
  • MLUserDataEncryption structure
  • ColumnImportance structure
  • Operations
  • CreateMLTransform action (Python: create_ml_transform)
  • UpdateMLTransform action (Python: update_ml_transform)
  • DeleteMLTransform action (Python: delete_ml_transform)
  • GetMLTransform action (Python: get_ml_transform)
  • GetMLTransforms action (Python: get_ml_transforms)
  • ListMLTransforms action (Python: list_ml_transforms)
  • StartMLEvaluationTaskRun action (Python: start_ml_evaluation_task_run)
  • StartMLLabelingSetGenerationTaskRun action (Python: start_ml_labeling_set_generation_task_run)
  • GetMLTaskRun action (Python: get_ml_task_run)
  • GetMLTaskRuns action (Python: get_ml_task_runs)
  • CancelMLTaskRun action (Python: cancel_ml_task_run)
  • StartExportLabelsTaskRun action (Python: start_export_labels_task_run)
  • StartImportLabelsTaskRun action (Python: start_import_labels_task_run)
• Data Quality API
  • Data types
  • DataSource structure
  • DataQualityRulesetListDetails structure
  • DataQualityTargetTable structure
  • DataQualityRulesetEvaluationRunDescription structure
  • DataQualityRulesetEvaluationRunFilter structure
  • DataQualityEvaluationRunAdditionalRunOptions structure
  • DataQualityRuleRecommendationRunDescription structure
  • DataQualityRuleRecommendationRunFilter structure
  • DataQualityResult structure
  • DataQualityRuleResult structure
  • DataQualityResultDescription structure
  • DataQualityResultFilterCriteria structure
  • DataQualityRulesetFilterCriteria structure
  • Operations
  • StartDataQualityRulesetEvaluationRun action (Python: start_data_quality_ruleset_evaluation_run)
  • CancelDataQualityRulesetEvaluationRun action (Python: cancel_data_quality_ruleset_evaluation_run)
  • GetDataQualityRulesetEvaluationRun action (Python: get_data_quality_ruleset_evaluation_run)
  • ListDataQualityRulesetEvaluationRuns action (Python: list_data_quality_ruleset_evaluation_runs)
  • StartDataQualityRuleRecommendationRun action (Python: start_data_quality_rule_recommendation_run)
  • CancelDataQualityRuleRecommendationRun action (Python: cancel_data_quality_rule_recommendation_run)
  • GetDataQualityRuleRecommendationRun action (Python: get_data_quality_rule_recommendation_run)
  • ListDataQualityRuleRecommendationRuns action (Python: list_data_quality_rule_recommendation_runs)
  • GetDataQualityResult action (Python: get_data_quality_result)
  • BatchGetDataQualityResult action (Python: batch_get_data_quality_result)
  • ListDataQualityResults action (Python: list_data_quality_results)
  • CreateDataQualityRuleset action (Python: create_data_quality_ruleset)
  • DeleteDataQualityRuleset action (Python: delete_data_quality_ruleset)
  • GetDataQualityRuleset action (Python: get_data_quality_ruleset)
  • ListDataQualityRulesets action (Python: list_data_quality_rulesets)
  • UpdateDataQualityRuleset action (Python: update_data_quality_ruleset)
• Sensitive data detection API
  • Using Sensitive Data Detection outside AWS Glue Studio
  • Data types
  • CustomEntityType structure
  • Operations
  • CreateCustomEntityType action (Python: create_custom_entity_type)
  • DeleteCustomEntityType action (Python: delete_custom_entity_type)
  • GetCustomEntityType action (Python: get_custom_entity_type)
  • BatchGetCustomEntityTypes action (Python: batch_get_custom_entity_types)
  • ListCustomEntityTypes action (Python: list_custom_entity_types)
• Tagging APIs in AWS Glue
  • Data types
  • Tag structure
  • Operations
  • TagResource action (Python: tag_resource)
  • UntagResource action (Python: untag_resource)
  • GetTags action (Python: get_tags)
• Common data types
  • Tag structure
  • DecimalNumber structure
  • ErrorDetail structure
  • PropertyPredicate structure
  • ResourceUri structure
  • ColumnStatistics structure
  • ColumnStatisticsError structure
  • ColumnError structure
  • ColumnStatisticsData structure
  • BooleanColumnStatisticsData structure
  • DateColumnStatisticsData structure
  • DecimalColumnStatisticsData structure
  • DoubleColumnStatisticsData structure
  • LongColumnStatisticsData structure
  • StringColumnStatisticsData structure
  • BinaryColumnStatisticsData structure
  • String patterns
• Exceptions
  • AccessDeniedException structure
  • AlreadyExistsException structure
  • ConcurrentModificationException structure
  • ConcurrentRunsExceededException structure
  • CrawlerNotRunningException structure
  • CrawlerRunningException structure
  • CrawlerStoppingException structure
  • EntityNotFoundException structure
  • GlueEncryptionException structure
  • IdempotentParameterMismatchException structure
  • IllegalWorkflowStateException structure
  • InternalServiceException structure
  • InvalidExecutionEngineException structure
  • InvalidInputException structure
  • InvalidStateException structure
  • InvalidTaskStatusTransitionException structure
  • JobDefinitionErrorException structure
  • JobRunInTerminalStateException structure
  • JobRunInvalidStateTransitionException structure
  • JobRunNotInTerminalStateException structure
  • LateRunnerException structure
  • NoScheduleException structure
  • OperationTimeoutException structure
  • ResourceNotReadyException structure
  • ResourceNumberLimitExceededException structure
  • SchedulerNotRunningException structure
  • SchedulerRunningException structure
  • SchedulerTransitioningException structure
  • UnrecognizedRunnerException structure
  • ValidationException structure
  • VersionMismatchException structure