View a markdown version of this page

Create rulesets - Amazon SageMaker Unified Studio
Navigate to the Data quality tabCreate a rulesetWrite rules using DQDLMigrate existing rulesets to your project

Create rulesets

To create a ruleset, first navigate to the Data quality tab for a catalog table, then define your rules using DQDL syntax.

Navigate to the Data quality tab

  1. In the left navigation pane, choose Data.

  2. Under the Catalogs tab, expand a catalog (for example, AwsDataCatalog), then expand the database (for example, sagemaker_sample_db).

  3. Select the table you want to evaluate (for example, churn).

  4. On the table detail page, choose the Data quality tab.

Create a ruleset

When you first open the Data quality tab on a table, it shows an empty state with the option to create your first ruleset.

  1. Choose Create.

  2. The Create ruleset page opens with the following fields: Ruleset name, Description (Optional), and the DQDL rules editor.

    Create ruleset page with DQDL editor

  3. Enter a name for your ruleset. The name identifies this set of rules in run results, the dashboard, and scheduled evaluations. Choose a descriptive name that reflects what the rules check (for example, completeness-checks or value-range-validation).

  4. In the DQDL rules section, use the code editor to define your rules. A Rule types reference panel lists all 31 available rule types. You can search rules by name and filter by category. Expand any rule type to see its description and syntax.

  5. After defining your rules, choose Create ruleset.

Write rules using DQDL

Define rules in the code editor using DQDL syntax. The editor provides autocomplete for rule types and column names from the table schema.

Tip

Use Ctrl+Tab (or Cmd+Tab on Mac) to trigger column name suggestions.

The following example shows a ruleset with two rules.

Rules = [
    RowCount > 1000,
    Completeness "account_length" between 0.1 and 0.25
]

Common rule types include:

Rule type What it checks
RowCount Total row count meets a threshold.
Completeness Percentage of non-null values in a column.
ColumnExists A specific column is present in the table.
ColumnLength String column values meet a length condition.
IsUnique All values in a column are unique.
IsComplete A column has no null values.
ColumnValues Column values satisfy a condition (for example, greater than 0, in a set).
CustomSql A custom SQL expression evaluates to true.

You can also use referential integrity rules to validate relationships between columns in different tables. For details on syntax and examples, see ReferentialIntegrity in the AWS Glue documentation.

Note

Referential integrity and other cross-table rule types are supported only for tables under the default AWS Glue Data Catalog, using the format <database>.<table>.

For the complete reference of all 31 rule types, expressions, composite rules, and syntax, see DQDL rule types in the AWS Glue documentation.

Migrate existing rulesets to your project

If you have existing rulesets in AWS Glue that you want to view and manage within your Amazon SageMaker Unified Studio project, tag them so the project can discover them.

  1. Add the following tag to your existing ruleset:

    • Key: AmazonDataZoneProject

    • Value: Your project ID (found in the project settings page in Amazon SageMaker Unified Studio)

  2. Ensure your project role has adequate permissions to access the underlying data, including access to the Amazon S3 bucket where the table data is stored and appropriate Lake Formation grants for the table.

Once tagged, the ruleset appears on the Data quality tab for the corresponding table within your project. You can then run, schedule, and view results the same way as rulesets created directly in Amazon SageMaker Unified Studio.