View a markdown version of this page

Data quality and data profiling notifications - Amazon SageMaker Unified Studio

Data quality and data profiling notifications

Data quality and data profiling notifications tell you about the outcome of data quality evaluations and data profile runs on tables in your catalog. For example, you can be notified when an evaluation score drops below a threshold that you set, or when a profile run detects anomalies.

In the schedule of a data quality ruleset or a data profile, any project member can choose which outcomes send notifications. These settings are saved on the ruleset.

Who receives data quality notifications

When a ruleset has notifications turned on, notifications go to all project members of the project where the notification settings were saved. This is the project in which you schedule the ruleset or data profile.

This includes users who are project members through a group.

The same recipients receive notifications for a profiling configuration for a catalog or database and for scheduled runs.

Events that send notifications

Data quality rulesets

Option in the schedule When it notifies Notification title
Evaluation run fails to execute The evaluation run could not complete. Data quality evaluation job execution error
Evaluation passes All rules passed, with a score of 100%. Data quality ruleset evaluation passed
Evaluation fails when the score is below [N]% The evaluation score is lower than the threshold. A score equal to the threshold doesn't send a notification. Data quality ruleset evaluation failed

Data profiles

Option in the schedule When it notifies Notification title
Profiling run completes A profile run finished. Data profiling run completed
Profiling run fails to execute A profile run could not complete. Data profiling run failed (event type Data Quality Ruleset Evaluation Error; see Notification types)
Profiling run detects anomalies A profile run found values that deviate from the expected pattern. Anomalies detected in data profile

These notification options apply to every run of the ruleset, including runs that you start manually.

Configure notifications for a data quality ruleset

To turn on notifications for a data quality ruleset
  1. In the catalog, open the table and choose the Data quality tab.

  2. Choose the Actions menu on your ruleset and select Schedule run. If the ruleset already has a schedule, open the schedule to edit it.

  3. Configure the schedule. For details, see Schedule evaluations.

  4. In the Notifications section, select the outcomes that you want to be notified about. By default, a new schedule has Evaluation run fails to execute and Evaluation fails when the score is below [N]% selected, with a threshold of 100%. If you select Evaluation fails when the score is below, you can change the threshold to any whole number from 1 to 100.

    The Notifications section of a data quality ruleset schedule, with options to notify when an evaluation fails to execute, passes, or falls below a score threshold.
  5. Choose Create schedule (or Save when you edit).

To turn on notifications for a data profile
  1. In the catalog, open the table and choose the Data profile tab.

  2. Choose the calendar icon or Edit Scheduled Run.

    The Run data profile panel opens.

  3. In the Schedule section, under Notifications, select one or more of Profiling run completes, Profiling run fails to execute, and Profiling run detects anomalies. By default, only Profiling run detects anomalies is selected for a new data profile schedule.

    The Notifications options in the Run data profile panel: Profiling run completes, Profiling run fails to execute, and Profiling run detects anomalies.
  4. Choose Save.

Notifications for a profiling configuration for a catalog or database

A profiling configuration for a catalog or database profiles every table in the catalog or database, including tables added later. You choose its notifications in the Schedule section under Notifications. The options apply to the profiling runs of every table that the configuration covers, and each notification names the table that it is about.

Option When it notifies
Profiling run completes A profiling run of a covered table finished.
Profiling run fails to execute A profiling run of a covered table could not complete.
Profiling run detects anomalies A profiling run of a covered table found values that deviate from the expected pattern.
To turn on notifications for a profiling configuration for a catalog or database
  1. In your project, choose Data in the left navigation pane to open the data explorer. On the Catalogs tab, select a catalog or a database in the tree.

  2. Choose the Data profile tab.

  3. Choose Create profiling configuration.

    The Create profiling configuration panel opens. Its subtitle shows Catalog: or Database: and the name. The panel is the same for a catalog and a database.

    The Create profiling configuration panel for a catalog or database, showing the Schedule section.
  4. In the Schedule section, under Notifications, select one or more of Profiling run completes, Profiling run fails to execute, and Profiling run detects anomalies. By default, only Profiling run detects anomalies is selected.

    The Notifications options for a catalog or database profiling configuration: Profiling run completes, Profiling run fails to execute, and Profiling run detects anomalies.
  5. Choose Create.

To change the notifications of an existing profiling configuration, on the Data profile tab choose Edit profiling configuration, change the options under Notifications, and then choose Save changes.