View a markdown version of this page

Profiling for catalogs and databases - Amazon SageMaker Unified Studio

Profiling for catalogs and databases

Instead of setting up profiling for each table one at a time, you can create a profiling configuration for an entire catalog or database. Amazon SageMaker Unified Studio then profiles every supported table in the catalog or database on the schedule you set, including tables that are added later. Results appear on each table's Data profile tab, with the same statistics, trend charts, and anomaly detection as profiles you run on a single table.

Profiling for catalogs and databases uses AWS Glue Data Quality evaluation configurations with a ruleset of analyzers that Amazon SageMaker Unified Studio creates for you. The analyzers gather statistics without pass or fail rules. For details about how evaluation configurations work, including scope, triggers, quotas, and cost, see Data quality monitoring at the catalog and database level in the AWS Glue Developer Guide.

Use a catalog or database configuration when you want consistent profiling across many tables. For example, you might profile every table in a database every night so that changes in row counts or completeness are flagged as anomalies. Anomaly detection starts flagging unexpected values after a table has been profiled at least three times.

Supported tables

You can create profiling configurations on AWS Glue Data Catalog tables, including Apache Hive and Apache Iceberg tables on Amazon S3, Apache Iceberg tables in Amazon S3 Tables, and Amazon Athena views. Federated catalogs and Amazon Redshift managed catalogs are not supported. For the full list, see Supported tables.

How profiling configurations are applied

You can set a profiling configuration on a catalog, a database, or a table. A table can be in scope of configurations at more than one level. When this happens, the most specific configuration applies:

  • A table-level configuration takes precedence over database-level and catalog-level configurations for that table.

  • A database-level configuration takes precedence over a catalog-level configuration for the tables that it includes.

The database-level or catalog-level configuration continues to profile all other tables in its scope. For example, you can profile every table in a catalog daily, and profile one critical table hourly. For more information, see When configurations overlap in the AWS Glue Developer Guide.

A paused table-level or database-level configuration still takes precedence over the catalog-level configuration for its tables. When a database has no configuration of its own and inherits one from its catalog, an information banner on the database's Data profile tab shows where the configuration is set. When you create a profile for a table that inherits a configuration, the profile form asks whether you want to override the inherited configuration for that table.

Create a profiling configuration for a catalog or database

To create a profiling configuration

  1. Open your project in Amazon SageMaker Unified Studio.

  2. In the left navigation pane, choose Data.

  3. Choose the catalog or database that you want to profile.

  4. Choose the Data profile tab, then choose Create profiling configuration.

  5. Configure the profiling scope:

    • Table scope: choose All tables, or enter a table name pattern to profile only matching tables.

    • Row scope: profile all rows, a percentage of rows, or the output of a SELECT query.

    • (Optional) Select Include advanced profiling statistics to also compute uniqueness and distinct value counts. Including advanced statistics increases run time and compute cost, especially for columns with many distinct values.

    • (Optional) Select Run profiling when table metadata changes are detected to also profile a table when its metadata changes. For what counts as a metadata change, see Triggers in the AWS Glue Developer Guide. For Apache Iceberg tables that are written to often, use a schedule instead to control cost.

    Note

    Value distributions, and the minimum and maximum values of timestamp columns, aren't available for catalog or database configurations.

  6. Configure the schedule: Choose the frequency and timezone for profile runs. Profiling configurations can run at most once an hour. For Status, choose Active to start profiling, or Paused to save the configuration without running it.

  7. (Optional) Configure notifications. For more information, see Notifications for a profiling configuration for a catalog or database.

  8. Choose Create.

Tables in the catalog or database are profiled on the next scheduled run. Tables that are added later are profiled when they're created. Each table is profiled at most once an hour across all configurations. Results appear on each table's Data profile tab as each run finishes.

Tables that already have a profile schedule

If a table already has its own profile schedule, Amazon SageMaker Unified Studio converts it to a table-level profiling configuration when you save. The table keeps its schedule and it takes priority over the catalog or database configuration.

Edit, pause, or delete a profiling configuration

  1. On the Data page, choose the catalog or database, then choose the Data profile tab.

  2. Do one of the following:

    • To change the configuration, choose Edit, update the settings, and then choose Save.

    • To stop runs without deleting the configuration, open the More actions menu (three dots) and choose Pause. To start again, choose Resume.

    • To delete the configuration, open the More actions menu and choose Delete, then confirm. Results from earlier profile runs stay on each table's Data profile tab, and its tables fall back to the next configuration level up.

Permissions

Whether you can create, edit, pause, or delete a catalog or database configuration depends on the IAM permissions of the project's role.

If you use AWS Lake Formation, the project's role profiles only the tables that it has permission to describe. For tables whose data Lake Formation manages, such as Amazon S3 Tables, the role also needs permission to select the data. For details, see Data quality monitoring at the catalog and database level.