

# Profiling for catalogs and databases

Instead of setting up profiling for each table one at a time, you can create a profiling configuration for an entire catalog or database. Amazon SageMaker Unified Studio then profiles every supported table in the catalog or database on the schedule you set, including tables that are added later. Results appear on each table's **Data profile** tab, with the same statistics, trend charts, and anomaly detection as profiles you run on a single table.

Profiling for catalogs and databases uses AWS Glue Data Quality evaluation configurations with a ruleset of analyzers that Amazon SageMaker Unified Studio creates for you. The analyzers gather statistics without pass or fail rules. For details about how evaluation configurations work, including scope, triggers, quotas, and cost, see [Data quality monitoring at the catalog and database level](https://docs.aws.amazon.com/glue/latest/dg/data-quality-catalog-monitoring.html) in the *AWS Glue Developer Guide*.

Use a catalog or database configuration when you want consistent profiling across many tables. For example, you might profile every table in a database every night so that changes in row counts or completeness are flagged as anomalies. Anomaly detection starts flagging unexpected values after a table has been profiled at least three times.

## Supported tables
<a name="dq-profile-supported-tables"></a>

You can create profiling configurations on AWS Glue Data Catalog tables, including Apache Hive and Apache Iceberg tables on Amazon S3, Apache Iceberg tables in Amazon S3 Tables, and Amazon Athena views. Federated catalogs and Amazon Redshift managed catalogs are not supported. For the full list, see [Supported tables](https://docs.aws.amazon.com/glue/latest/dg/data-quality-catalog-monitoring.html#data-quality-catalog-monitoring-supported-tables).

## How profiling configurations are applied
<a name="dq-profile-config-precedence"></a>

You can set a profiling configuration on a catalog, a database, or a table. A table can be in scope of configurations at more than one level. When this happens, the most specific configuration applies:
+ A table-level configuration takes precedence over database-level and catalog-level configurations for that table.
+ A database-level configuration takes precedence over a catalog-level configuration for the tables that it includes.

The database-level or catalog-level configuration continues to profile all other tables in its scope. For example, you can profile every table in a catalog daily, and profile one critical table hourly. For more information, see [When configurations overlap](https://docs.aws.amazon.com/glue/latest/dg/data-quality-catalog-monitoring.html) in the *AWS Glue Developer Guide*.

A paused table-level or database-level configuration still takes precedence over the catalog-level configuration for its tables. When a database has no configuration of its own and inherits one from its catalog, an information banner on the database's **Data profile** tab shows where the configuration is set. When you create a profile for a table that inherits a configuration, the profile form asks whether you want to override the inherited configuration for that table.

## Create a profiling configuration for a catalog or database
<a name="dq-profile-create-config"></a>

**To create a profiling configuration**  


1. Open your project in Amazon SageMaker Unified Studio.

1. In the left navigation pane, choose **Data**.

1. Choose the catalog or database that you want to profile.

1. Choose the **Data profile** tab, then choose **Create profiling configuration**.

1. Configure the profiling scope:
   + **Table scope:** choose **All tables**, or enter a table name pattern to profile only matching tables.
   + **Row scope:** profile all rows, a percentage of rows, or the output of a SELECT query.
   + (Optional) Select **Include advanced profiling statistics** to also compute uniqueness and distinct value counts. Including advanced statistics increases run time and compute cost, especially for columns with many distinct values.
   + (Optional) Select **Run profiling when table metadata changes are detected** to also profile a table when its metadata changes. For what counts as a metadata change, see [Triggers](https://docs.aws.amazon.com/glue/latest/dg/data-quality-catalog-monitoring.html#data-quality-catalog-monitoring-triggers) in the *AWS Glue Developer Guide*. For Apache Iceberg tables that are written to often, use a schedule instead to control cost.
**Note**  
Value distributions, and the minimum and maximum values of timestamp columns, aren't available for catalog or database configurations.

1. Configure the schedule: Choose the frequency and timezone for profile runs. Profiling configurations can run at most once an hour. For **Status**, choose **Active** to start profiling, or **Paused** to save the configuration without running it.

1. (Optional) Configure notifications. For more information, see [Notifications for a profiling configuration for a catalog or database](data-quality-notifications.md#data-quality-notifications-catalog-config).

1. Choose **Create**.

Tables in the catalog or database are profiled on the next scheduled run. Tables that are added later are profiled when they're created. Each table is profiled at most once an hour across all configurations. Results appear on each table's **Data profile** tab as each run finishes.

### Tables that already have a profile schedule
<a name="dq-profile-existing-schedule"></a>

If a table already has its own profile schedule, Amazon SageMaker Unified Studio converts it to a table-level profiling configuration when you save. The table keeps its schedule and it takes priority over the catalog or database configuration.

## Edit, pause, or delete a profiling configuration
<a name="dq-profile-edit-config"></a>

1. On the **Data** page, choose the catalog or database, then choose the **Data profile** tab.

1. Do one of the following:
   + To change the configuration, choose **Edit**, update the settings, and then choose **Save**.
   + To stop runs without deleting the configuration, open the **More actions** menu (three dots) and choose **Pause**. To start again, choose **Resume**.
   + To delete the configuration, open the **More actions** menu and choose **Delete**, then confirm. Results from earlier profile runs stay on each table's **Data profile** tab, and its tables fall back to the next configuration level up.

## Permissions
<a name="dq-profile-config-permissions"></a>

Whether you can create, edit, pause, or delete a catalog or database configuration depends on the IAM permissions of the project's role.

If you use AWS Lake Formation, the project's role profiles only the tables that it has permission to describe. For tables whose data Lake Formation manages, such as Amazon S3 Tables, the role also needs permission to select the data. For details, see [Data quality monitoring at the catalog and database level](https://docs.aws.amazon.com/glue/latest/dg/data-quality-catalog-monitoring.html).