View a markdown version of this page

Data quality evaluation - Amazon SageMaker AI

Data quality evaluation

What it is

With data quality evaluation, you measure the quality of a text or tabular dataset before you use it to train or evaluate a model. You run a job on a dataset that is registered in your hub and choose the evaluations in the job's evaluation configuration. SageMaker AI reports the results for the whole dataset and for each row. The job reads the dataset and never modifies it.

The job writes an aggregate report for the whole dataset and a row-level report, a copy of your rows with results that the job adds as columns. Some evaluations write more files, such as the findings file of the exploratory analysis.

You can evaluate dataset quality in three ways, and you can combine them in one job:

  • Built-in metrics: metrics that the service provides and that you choose by name, such as word_count or pii_detection, for length and format, duplicates, personally identifiable information (PII), language, topics, and diversity.

  • Data quality rules: Boolean conditions that each row must pass. A rule can check the fields of each row or the dataset as a whole. You write your rules in a data quality specification file, which can also ask for a label or a score for each row. A large language model (LLM) can judge a rule with a prompt that you write, a custom LLM-as-a-judge criterion, on its own or against documents that you provide.

  • Exploratory analysis: an agent that reads the dataset and writes the problems it finds, each with a title and the rows that show it, to a findings file, for a dataset where you do not yet know what to check. To act on the findings, see Reading the results for a decision.

Only data quality rules use a specification file. You choose built-in metrics and exploratory analysis by name in the job's evaluation configuration. Data quality rules are the recommended way, because they check your rows against conditions that you define and name the rules that each row failed.

When to use it

Use data quality evaluation when:

  • You want to check the quality of a dataset that you generated with synthetic data generation (SDG) before you use it for training or customization.

  • You want to bring your own dataset (BYOD) for training and have its format validated and its quality measured first.

  • You want every row checked against data quality rules that you have written.

  • You want to check for duplicate rows.

  • You want the dataset explored for problems, because you do not yet know what to check.

  • You want to know which topics the dataset covers.

How it works

A data quality evaluation is a data preparation job with the job category DataQualityEvaluation, created with the same CreateJob operation as the other data preparation jobs.

The job

A job has five parts:

  • The job name, a name that no existing job has.

  • The execution role, the IAM role that the job uses to read the dataset and write its files.

  • The input dataset, one registered dataset.

  • The output location, an Amazon S3 location in the job's AWS Region.

  • The evaluation configuration, the list of operations to run, each an operator and its settings.

The dataset

A job evaluates one dataset, named by its ARN in InputDataConfig. The dataset is hub content of type DataSet in a hub in your account. Its registry entry names the file format in DatasetFileFormat, which decides the columns that the operators read when you name none.

The job lifecycle

CreateJob returns the job ARN. DescribeJob reports a JobStatus and a SecondaryStatus with a message for each status the job passed through. The JobStatus is InProgress, Stopping, Completed, Failed, or Stopped. The secondary statuses of a successful job, in order, are the following.

Secondary status What the job is doing
Validating Validates the Region, the Amazon S3 locations, and the data quality specification or code artifact named for the job.
Preparing Prepares the evaluation artifacts.
Executing Executes the operations.
Publishing Writes the output files to the output location.
Finalizing Adds the job's entry to the dataset's registry entry.
Completed Done, with MetadataS3Uri, the location of the job's metadata.json, inside the returned configuration of the DescribeJob response.

A Failed job has a FailureReason. A job that has reached Publishing can no longer be stopped. A stopped job has no FailureReason, writes nothing under its own folder, and adds no entry to the dataset.

The output location

Every job writes to its own folder in the output location, so several jobs can share one S3OutputPath. The job also records its evaluation on the dataset: it adds an entry that points to its output to the JobMetadata list of the dataset's entry in your hub. It creates no new dataset and no new version, so the dataset's ARN and version do not change.

The operations

EvaluationConfig.Operations lists the operations of the job. Each operation names an Operator, a built-in metric or an evaluation from the table under Operators, and an optional Config with its settings. The following operation counts the words of the prompt column. In the following example, replace prompt with the name of your text column:

{"Operator": "word_count", "Config": {"View": [{"Role": "Text", "Column": "prompt"}]}}

Config.View names the columns that the operator reads, one entry for each column, each a role and a column name. The roles are the following:

  • Text, a column of text, read by the text metrics.

  • Column, a numeric or categorical column, read by the column statistics, with one result for each column.

  • Label, a label column, read by the balance metrics.

  • Messages, a column of conversation turns.

The other keys of Config are the operator's settings, and the service refuses a key that the operator does not define when you submit the job. The following operation detects the language of the prompt column and flags the rows that are not English, with the accepted languages and the lowest accepted score as settings:

{"Operator": "language_detection", "Config": {"View": [{"Role": "Text", "Column": "prompt"}], "languages": ["en"], "min_score": 0.5}}

Without a View, an operator reads the default text columns of the dataset's format: the prompt column and, where the format has one, the completion column. You can run the same operator on two columns as two operations in one job. The service refuses the same operator with the same Config twice as a duplicate.

Operators

An operator is a measurement, a check, or an analysis that the job runs on your dataset. Every built-in metric writes its summary to the aggregate report, and some also write a column for each row to the row-level report. The following table lists each operator group, what it does, the columns it reads, and its operators: 41 built-in metrics and the four evaluations default_metrics, spec_eval, exploratory_analysis, and data-validation.

Operator group What it does Columns read Operator
Data quality rules Evaluation of every row, or a sample, and the whole dataset against the rules in your data quality specification file. The whole row. spec_eval
Exploratory analysis An agent that explores the dataset and writes its findings. The whole row. exploratory_analysis
Text quality The length, format, and variety of the text. effectiveness: one score for the size of the dataset, no column read. One text column, role Text. word_count, character_count, compression_ratio, format_distribution, plain_text_ratio, content_structures, uniqueness, ngram_diversity, mif_score, effectiveness
Text filters Flags on the rows to filter out, with no row removed: text that is mostly URLs, text with few letters and digits, repeated words or characters, broken character encoding, text that is too short or too long, and lines that are too short. One text column, role Text. urls_filter, alphanumeric_filter, word_repetition_filter, char_repetition_filter, mojibake_filter, character_length_filter, average_line_length_filter
Keyword topics The rows for each topic from a keyword list that you provide, and the share of your topics that the dataset covers. One text column, role Text. keyword_topic_counts, keyword_topic_coverage
Column statistics A statistic for each numeric column, and the counts of null values, non-numeric values, the most frequent value, and the frequency of each value. One or more columns, role Column. min, max, mean, median, std_dev, skewness, kurtosis, percentiles, count_zero, count_null, count_non_numeric, mode, value_frequency
Label balance How evenly the label values are distributed, as a balance score and as a ratio, over all labels and for each label column. One or more columns, role Label. data_balance, data_balance_ratio, field_balance, field_balance_ratio
Language The language of each row, and a flag on the rows whose language is not in your list. One text column, role Text, required on every format. language_detection
PII Flags on the sampled rows that contain personally identifiable information (PII). One column, role Text or Messages, required on every format. pii_detection
Duplicates Flags on the rows whose text repeats another row's, byte for byte or by similarity, with no row removed. One text column, role Text. exact_dedup, fuzzy_dedup
Taxonomy A three-level taxonomy of the rows for each framing that you request, task, domain, or quality. One text column, role Text. taxonomy
Default metrics Nine of the text quality metrics at once on the format's default text columns. The format's default text columns. default_metrics
Format validation Every record checked against the file format that you declared when you registered the dataset. The whole record. data-validation

Metric settings

The following built-in metrics take settings in Config, beside the View.

Operator Setting Values
uniqueness CaseSensitive true by default.
ngram_diversity N 1 to 8, with 1 the default.
mif_score mode corpus or per_row.
character_length_filter min_length, max_length The shortest and the longest accepted text, in characters.
keyword_topic_counts, keyword_topic_coverage topic_keywords A map from each topic to its keywords.
keyword_topic_counts, keyword_topic_coverage default_topic The topic for the rows that no keyword matches.
language_detection languages A list of language codes.
language_detection min_score The lowest accepted score for the detected language, with the rows below it flagged.
taxonomy framings A list from task, domain, and quality, with all three the default.
exact_dedup Normalize false by default.
fuzzy_dedup JaccardThreshold The similarity at which two rows count as duplicates, with 0.7 the default.
fuzzy_dedup Lowercase true by default.

Data quality specification evaluation

spec_eval evaluates each row, or a sample of the rows, and the whole dataset against the data quality specification file named in its Config. It reads whole rows and takes no View. Its Config takes the following keys.

Key What it is Required
SpecS3Uri The Amazon S3 location of your data quality specification file, a .md or .txt file in the prose or the structured form. Exactly one of SpecS3Uri and CodeArtifactS3Uri.
CodeArtifactS3Uri The spec_eval/ folder that an earlier job wrote, as it is or with a judge_code.py that you modified. The job runs that code and reads no specification file. Exactly one of SpecS3Uri and CodeArtifactS3Uri.
ContextDocumentsS3Uri An Amazon S3 folder of context documents that your criteria are judged against. The folder also adds the service's own faithfulness check. Optional.
DownSampleRecordCount The number of rows to judge instead of all of them. The rows outside the sample carry null in the four specification columns of the row-level report. Optional.
ModelConfig.JudgeModelConfig.ModelId The model that judges the rows, us.anthropic.claude-sonnet-5 by default. The accepted ids are us.anthropic.claude-sonnet-5, us.anthropic.claude-sonnet-4-6, us.anthropic.claude-opus-4-6-v1, us.anthropic.claude-sonnet-4-5-20250929-v1:0, us.anthropic.claude-sonnet-4-20250514-v1:0, us.anthropic.claude-haiku-4-5-20251001-v1:0, and openai.gpt-oss-120b-1:0, and the service refuses another id. Optional.

Exploratory analysis

exploratory_analysis runs an agent that reads the dataset and writes a list of findings, each with a title and the rows that show it, to the findings file aggregate/exploratory/findings.json. It takes no Config and writes no row column.

PII detection

pii_detection uses Amazon Bedrock models to find personally identifiable information (PII). It reads a sample of the rows of one Text or Messages column, so its View has exactly one entry and is required on every format. It flags each sampled row that contains PII and records the kinds found. It takes no other setting and never writes a detected value to the output.

PII detection finds direct identifiers only: information that identifies an individual on its own, such as a name or an account number. It does not find quasi-identifiers: information that identifies an individual only in combination with other information.

The categories it detects are the following:

  • Names and contact details: private names, public names, email addresses, and phone numbers.

  • Addresses and locations: full addresses, partial addresses, and GPS coordinates.

  • Government and financial numbers: card numbers, card codes and expiry dates, bank account numbers, PINs, Social Security numbers, driver's license numbers, passport numbers, and other government IDs.

  • Account, device, and network identifiers: IP addresses, MAC addresses, user names, device and tracking IDs, and account and membership numbers.

  • Vehicle identifiers: license plates and vehicle identification numbers.

  • Passwords and credentials: passwords and other credentials.

  • Dates and age: personal dates, such as a date of birth, and age.

  • Web addresses: public URLs and private URLs.

Taxonomy

taxonomy groups the rows of one text column into a three-level taxonomy for each framing that you request, task, domain, or quality. It writes the three levels of each framing to the row-level report and an HTML report.

Deduplication

exact_dedup and fuzzy_dedup flag the duplicates in one text column and remove no row. Each flagged row carries a reason and the id of the first row of its group.

Default metrics and format validation

default_metrics runs word_count, character_count, format_distribution, plain_text_ratio, compression_ratio, uniqueness, ngram_diversity, mif_score, and effectiveness on the format's default text columns and takes no Config. On a format without default columns, list the metrics yourself with a View.

data-validation checks every record against the format that you declared, and it reads JSON Lines and Parquet datasets. It takes no Config and must be the only operation in its job.

How a data quality specification works

A data quality specification file states your data quality rules, the conditions that each row or the whole dataset must pass, and any measurements that you want taken. You write the rules in your own words, such as that a reply must answer the question or that a completion must have at least eight words. The job turns the file into criteria, runs them on every row or on a sample, and reports which rows failed which rule and the value of each measurement.

The two forms

You write a data quality specification in one of two forms:

  • The prose form: rules written as sentences, with no special syntax, for a first specification or for rules that are still changing.

  • The structured form: a ## Criteria section whose entries name each criterion and state how it is decided, for an evaluation that you repeat with stable criterion ids.

The ## Criteria heading is what makes a file structured, and both forms are read the same way whether the file name ends in .md or .txt.

Rules and measurements

Each rule or measurement that you write becomes a criterion, the unit that the job evaluates. A criterion is a rule, with a pass or fail verdict, or a measurement, with a value and no verdict, for every row or once for the whole dataset. From a prose file, a model derives one criterion for each rule or measurement and takes a threshold from the sentence when it states one. From a structured file, the job reads the ids and the descriptions as written.

How a criterion is decided

The job decides a criterion from your description, with your own prompt, or against your context documents, and a criterion can combine the last two. In the prose form the job chooses from your sentences, and in the structured form you choose with the keys of the entry:

  • From your description. The service turns the description into generated code or into a judge prompt for a model, whichever fits the criterion. This is the default.

  • With your own prompt. A prompt line sends your text to the model as written, in place of the judge prompt that the service would write. This is a custom LLM-as-a-judge criterion: the model judges each row with your prompt, as a rule or as a measurement.

  • Against your context documents. The documents that you give the job are the reference for the criterion. In the structured form a criterion marked context: documents is judged this way, and in the prose form a rule or a measurement that refers to the documents is judged this way.

The following structured data quality specification shows one criterion decided from its description, one judged with your prompt, and one judged with your prompt against the documents:

Example Data quality specification using three evaluation methods
# Replies judged three ways ## Criteria - id: completion_present description: The completion field is present and is not blank or whitespace only. - id: reply_is_actionable prompt: You are reviewing a customer-support reply. Read the PROMPT (the customer's message) and the COMPLETION (the agent's reply). Answer PASS if the reply tells the customer exactly what will happen next or what they must do next. Answer FAIL if the reply only acknowledges the problem, restates it, or gives no next step. description: The reply tells the customer what happens next or what to do next. - id: reply_follows_guide context: documents prompt: You are reviewing a customer-support reply against the reference documents. Read the COMPLETION (the agent's reply) and the documents. Answer PASS if the reply follows every rule that the documents state for replies. Answer FAIL if it breaks one, and name the rule. description: The reply follows the rules in the style guide.

The service decides completion_present from its description, judges reply_is_actionable with your prompt, and judges reply_follows_guide with your prompt against the documents.

Context documents

You can give the job context documents in an Amazon S3 folder, named by ContextDocumentsS3Uri in the spec_eval operation. When you do, the service adds its own check, builtin_faithfulness, which judges whether each row is grounded in the documents: every factual claim in the row must be stated by the documents or follow from them. A row that fails this check counts as failed, so give the job documents that state the facts that your rows rely on. The pass rate of a criterion judged against the documents, the service's own or yours, is under context_criteria in the aggregate report.

Important

The data quality specification and the context documents control how your rows are judged. The job sends your specification, and your context documents when you give them, to the model as written, and the service does not detect or remove instructions in that text. The model also follows text that you did not write as a rule, such as instructions inside a document that you downloaded, and such text can change the verdicts. Before you submit a job, review the specification and the documents, and use only files that you wrote or that you trust.

What the job writes

A spec_eval operation writes the following to the output folder, the same for both forms:

  • The criteria that the job derived or read, in spec_eval/extracted_criteria.json in the output folder, so that you can check how it read your rules.

  • A spec_eval entry in the aggregate report, with the counts and the pass rates over the dataset.

  • Four columns for each row in the row-level report: the verdict, the rules that the row failed, the criteria that could not be evaluated, and each criterion's result.

Data quality specification examples

Structured data quality specification example

The following structured data quality specification was run on the 200-row customer-support dataset of the Use example. It holds a rule decided by code, a rule decided by a model, a label, a score, and a rule over the whole dataset, five of the file's seven criteria, with the description of tone shortened. The table at the end of this section explains each key.

# Data quality rules for the customer-support training dataset ## Stratify by - category ## Criteria - id: completion_min_length type: code! threshold: {"min_words": 8} description: The completion contains at least min_words words, counted by splitting on whitespace. - id: answers_the_question type: llm! description: The completion answers what the customer asked in the prompt. A reply that restates the question, changes the subject, or refuses without a reason fails. - id: tone kind: metric values: [formal, friendly, curt] description: Classify the tone of the completion as formal, friendly, or curt. - id: helpfulness kind: metric values: 0 to 1 description: Score how completely the completion resolves the customer's problem, from 0.0 (does not help) to 1.0 (fully resolves it), in steps of 0.25. 0.5 means a partial answer that leaves the customer with a next step to find on their own. - id: answer_rate scope: aggregate inputs: [answers_the_question] threshold: {"min_rate": 0.9} description: At least min_rate of the rows pass answers_the_question.

The job judged all 200 rows, of which 173 passed and 27 failed. completion_min_length had a pass rate of 0.96 and answers_the_question of 0.87, with a pass rate for each value of category, such as 0.84 on the billing rows. answer_rate returned one verdict, a fail, with the reason 174/200 rows passed answers_the_question (rate=0.870), threshold=0.9. The labels of tone and the scores of helpfulness appear per row as the value of their entries, with no pass rate and no distribution in the aggregate report.

Sections and keys of a structured data quality specification

The following table lists the sections and the keys of a structured data quality specification.

Section or key What it is
## Stratify by An optional section before ## Criteria with one column name on each line. Every row rule then reports its pass rate for each value of that column.
## Criteria The section that holds the criteria, one entry each, and the heading that makes the file structured.
id The name of the criterion, which starts its entry and keys its result in the report and in every row.
kind rule, the default, for a pass or fail verdict, or metric for a measurement with a value and no verdict.
type What decides the criterion: auto, the default, lets the job choose, code and llm request generated code or a model, and code! and llm! lock that choice, which the report records. On a criterion with a prompt or a context: documents line, leave the type line out, because the model always judges such a criterion.
scope row, the default, for a result per row, or aggregate for one result over the whole dataset.
values The results that a measurement may return, as labels in brackets such as [formal, friendly, curt] or as a range such as 0 to 1.
threshold A JSON object with the numbers that a rule compares against, such as {"min_words": 8}, on rules only.
inputs The row criteria that an aggregate criterion reads, such as the row rule whose pass rate it requires.
reads The fields of your rows that the criterion uses, such as [completion], echoed in the report. A field that your rows do not have makes the criterion not evaluable on any row.
context: documents The model judges the criterion against the documents that you provide. In the row-level report such a criterion has the source context, its result in outcome instead of verdict, such as fail, measured, not_applicable, or not_evaluable, and the passage it relied on in quote and evidence.
prompt The exact text that the model judges with, one paragraph for each line, sent as written in place of the judge prompt that the service writes from the description. The description stays for the report. With a context: documents line as well, the model judges with your prompt against the documents.
description What the criterion evaluates, in words, which the generated code or the model follows.

Inputs

A job takes one registered dataset, often the dataset that Synthetic data generation produced and registered as hub content of type DataSet, whose arn you pass to the job unchanged. You can also bring your own dataset (BYOD), register it in a hub, and evaluate it the same way. For data quality rules, you also need the data quality specification file and, if your criteria refer to documents, the context documents.

In the snippets on this page, replace amzn-s3-demo-bucket with the name of your Amazon S3 bucket.

The dataset file and its format

The job reads one dataset from one Amazon S3 location, a single file or a folder of Parquet shards with identical columns. The file's extension decides how it is read:

  • .jsonl: JSON Lines, one JSON object per line.

  • .csv: CSV with a header row that names every column.

  • .parquet: Parquet.

  • .sqlite or .db: a SQLite database with one table.

A file with any other extension fails the job. The job reads text and tabular data, not image, audio, or video data. The location must be in the job's Region and in your account. Column names that start with dq_input__ are reserved for the job's own derived columns. The dataset's files total at most 18 GB for a job that runs exploratory_analysis, 30 GB for a job that runs spec_eval, and 90 GB for every other job. A job with several operations takes the smallest of their limits.

When you register the dataset, you name its format in DatasetFileFormat, and most training formats give the job default text columns to read when an operation has no View. The row shape of each format is described in Datasets and hub content. Without a View, an operator reads the format's prompt column and, where the format has one, its completion column, such as prompt and completion for HF_Prompt_Completion. Nova_RLVR, Nova_DPO, MTRL, and a Custom dataset whose columns are not named prompt and completion have no default columns, so every operator needs a View on them. When your rows fit no named format, register the dataset as Custom and name the column that each operator reads in a View.

Registering your own data

To bring your own dataset (BYOD), upload the file to a prefix of its own in Amazon S3 and register that prefix as a dataset in a hub in your account. In the Amazon SageMaker Python SDK, DataSet.create registers the dataset and returns the DataSet whose arn you pass to the job. The registry entry it writes has the DatasetType BYOD, the DatasetFileFormat, and the data's location:

{ "DatasetType": "BYOD", "DatasetFileFormat": "HF_Prompt_Completion", "DatasetS3Uri": "s3://amzn-s3-demo-bucket/datasets/support-conversations/" }

A dataset that DataSet.create registers has the document schema that the job requires.

The data quality specification file

Upload your data quality specification file, with a name that ends in .md or .txt, to Amazon S3. To judge rows against your own documents, upload them to a folder of their own. You name both locations in the spec_eval operation, with SpecS3Uri and ContextDocumentsS3Uri. If the dataset came from synthetic data generation, you can pass the specification that generated it as SpecS3Uri, because a generation specification is also a valid specification for this job, which reads only the file that you name. Review the specification and the documents before you upload them, as the Important note under How a data quality specification works says.

Use data quality evaluation

You can run data quality evaluation in the Amazon SageMaker Python SDK directly, or through a coding assistant with the core AI and ML skill of the Agent Toolkit for AWS. Both run the same job and write the same result files.

Amazon SageMaker Python SDK

In the following example, you evaluate a dataset of 200 customer-support conversations prepared for training a reply model, with the customer's message in prompt and the agent's reply in completion. One job runs three operations:

  • The built-in default metrics.

  • The rules in your data quality specification file.

  • An exploratory analysis, for problems that your specification does not cover.

For another evaluation, change the Operations list.

The example assumes an execution role and a default Amazon S3 bucket, as described in Set up and prerequisites, and a registered dataset, as described under Inputs. The file support-spec.md is a structured specification in the form that How a data quality specification works shows. Upload it to Amazon S3 with the following AWS CLI command:

aws s3 cp support-spec.md s3://amzn-s3-demo-bucket/dq/support-spec.md

Then run the following Python code:

from sagemaker.core.helper.session_helper import Session from sagemaker.ai_registry import DataPreparer, DatasetJobConfig, JobCategory session = Session() preparer = DataPreparer( sagemaker_session=session, role="arn:aws:iam::123456789012:role/role-name", ) # 1. List every field that a data quality evaluation job accepts. DatasetJobConfig.print_schema(JobCategory.DataQualityEvaluation) # 2. Describe the job: the registered dataset, where to write the results, # and the three operations to run on every record. config = DatasetJobConfig( schema_version="3.0.0", # selects the request format this page documents input_data_config=[{ "DatasetArn": "arn:aws:sagemaker:region:123456789012:hub-content/" "hub-name/DataSet/support-conversations/0.0.1", }], output_data_config={ "S3OutputPath": "s3://amzn-s3-demo-bucket/dq/support-conversations/out/", }, evaluation_config={ "Operations": [ {"Operator": "default_metrics"}, {"Operator": "spec_eval", "Config": { "SpecS3Uri": "s3://amzn-s3-demo-bucket/dq/support-spec.md", }}, {"Operator": "exploratory_analysis"}, ], }, ) # 3. Submit the job and wait for it. The call prints the job ARN when it # submits, shows the job's status while it waits, and returns the dataset # with job_arn set when the job completes. dataset = preparer.evaluate(config, base_job_name="support-conversations-dq", wait=True) print(dataset.arn, dataset.job_arn) # 4. If your session ends while the job runs, reconnect with the job ARN # that step 3 printed. # dataset = DataPreparer.attach("job-arn")

The fields of the example are the following. The keys of the spec_eval operation are in its entry under Operators.

Field What it is Required
role on DataPreparer The execution role. Without it, the SDK reads the role from your SageMaker AI configuration or from get_execution_role(). Optional when a role resolves from your configuration, otherwise the SDK refuses to submit.
schema_version on DatasetJobConfig The request format that this page documents, in which EvaluationConfig holds an Operations list. Without it, the SDK sends a request format that does not accept Operations. Required for the request shown here.
InputDataConfig[].DatasetArn The ARN of the registered dataset to evaluate. Required.
OutputDataConfig.S3OutputPath The Amazon S3 prefix for the result files. Without it, the SDK fills s3://default-bucket/DataPreparer/DataQualityEvaluation/base-job-name. Required in the request, supplied by the SDK when you omit it.
EvaluationConfig.Operations[].Operator The operator to run. At least one operation.
base_job_name on evaluate() The prefix of the job name, to which the SDK appends a timestamp. To name the job yourself, pass job_name instead, a name that no existing job has. Optional.

The evaluate() call polls the job's status every 30 seconds and prints the first 10 rows of the row-level report when the job completes. The returned object carries the dataset and the job ARN and no results, which the job writes to the output location. To submit the job and return at once, pass wait=False, which returns a handle on the job, and later run DataPreparer.attach("job-arn") to wait for the job and receive its DataSet.

The job ARN has the form arn:aws:sagemaker:region:account-id:job/DataQualityEvaluation/job-name. You manage a submitted job with DescribeJob, ListJobs, StopJob (or DataPreparer.stop("job-arn") in the SDK), and DeleteJob, which the Amazon SageMaker AI API Reference documents. A deleted job's entry stays on the dataset.

Coding assistant with the AI and ML skill

With a coding assistant and the core AI and ML skill of the Agent Toolkit for AWS, the assistant runs data quality evaluation for you through a guided conversation. To install it, see the Agent Toolkit for AWS User Guide. As with the SDK, use an execution role and a default Amazon S3 bucket, as described in Set up and prerequisites.

Start with a prompt for the coding assistant that describes your data and the use you intend for it:

I have 200 customer-support conversations in s3://amzn-s3-demo-bucket/dq/support-conversations.jsonl that I want to train a reply model on. My data quality rules are in support-spec.md. Tell me whether the dataset passes those rules, and which records do not.

The assistant then works through the same steps as the SDK example, and asks for your approval before each job:

  1. The assistant gets your data quality rules, by asking what the data must satisfy and turning your answers into rules, or by reading the data quality specification that you already have.

  2. It checks the file's format and registers the dataset, then tells you which built-in metrics apply to that format.

  3. It shows you the plan, with the dataset, the specification, the metrics, the exploratory analysis, the output location, and the Region, and tells you that the job is billed to your account. Nothing is uploaded, registered, or submitted until you confirm.

  4. It submits the job and waits for it, or it saves the work as a notebook or script that you can run and rerun yourself.

  5. It reads the result files, writes a report with the findings, the records that failed your rules, and the tokens used, and offers the next steps, such as transforming the flagged records.

Permissions and billing

A data quality evaluation job needs the following permissions, whichever entry point submits it.

Who Permissions
The execution role bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream, s3:GetBucketLocation, read and write access to the input and output locations, and sagemaker:ImportHubContent on the hub that holds the dataset.
The identity that manages jobs sagemaker:DescribeJob and sagemaker:StopJob on the job ARN, sagemaker:DeleteJob, and sagemaker:ListJobs.
The identity that submits the job sagemaker:CreateJob on the job ARN, iam:PassRole for the execution role to job.sagemaker.amazonaws.com, and sagemaker:ImportHubContent and sagemaker:UpdateHubContent on the dataset and its hub, so that the job can record its result on the dataset.
The trust policy of the execution role sagemaker.amazonaws.com allowed to assume the role. This is a different service name from the one in the iam:PassRole condition above, and the job needs both as written.
Note

The job calls Amazon Bedrock models on your behalf, so the execution role needs permission to call them, and the models' pricing and on-demand quotas apply to the job. The following evaluations call a model:

  • The data quality specification evaluation calls a model once to derive the criteria from your specification. Then, for every row, it calls a model on each criterion that a model decides. Criteria that generated code decides cost no model calls. To try a specification on a sample first, set DownSampleRecordCount.

  • The exploratory analysis, PII detection, and taxonomy call a model on the rows they read.

The file token_usage_report.json in the output folder records the models the job called and the tokens it used.

Results

A completed job writes the following files under its own folder in the output location, S3OutputPathdata-quality-evaluation/job-name/created-at/. The following table lists each file, what it holds, and when the job writes it.

File What it holds When the job writes it
The index file metadata.json The starting point of a run: the job, the dataset, the evaluation configuration you sent, the time, and under Reports the locations of the aggregate report, the row-level report, and the findings and data quality specification evaluation files when the job writes them. Its own location is MetadataS3Uri on the dataset's entry and in the DescribeJob response. It does not list token_usage_report.json or the taxonomy report. Every completed job.
The aggregate report aggregate/quality_report.json One entry per measurement over the whole dataset. Every completed job except a data-validation job, which writes its own report under validation/.
The row-level report record_level/evaluated_data.jsonl A copy of your rows, unchanged, with the metric values and the rule verdicts added as columns whose names start with dq_. Every completed job, unless its only operation is exploratory_analysis or it is a data-validation job.
The usage file token_usage_report.json The models the job called and the tokens it used, in total and for each model. Every completed job.
The folder spec_eval/ extracted_criteria.json, the criteria the job derived or read. judge_code.py, the program that judged the rows. judge_config.json, the model id, the token cap, and the Region of the judging. reproduce.py, a script that replays the judging on your own machine. judge_texts.json, for a job that judged against your documents. A job that ran spec_eval.
The findings file aggregate/exploratory/findings.json The problems the analysis found, each with a title and evidence, the counts and example rows it rests on. The same list is under the exploratory_analysis key of the aggregate report. A job that ran exploratory_analysis.
The taxonomy report aggregate/taxonomy/run-id/taxonomy_report.html The taxonomy of the rows, where run-id is the run identifier the job assigns. Its location is taxonomy_report_path in the taxonomy entry of the aggregate report. A job that ran taxonomy.
The files under validation/ The validation set's metadata file, a report with the counts of valid and failed records, the failure rate, and the record shape the job detected, one line for each record that failed, and the indices of those records, beside the job's own metadata.json and token_usage_report.json. A data-validation job.

The columns that the operators write

The following table lists the columns that each operator writes to the row-level report, where <column> stands for the column that the operator read.

Metrics Row columns Notes
Text quality: word_count, character_count, compression_ratio, content_structures, and mif_score in its per_row mode One column for each metric, named after the column read, such as dq_word_count__prompt. The other five text quality metrics write report entries only.
Text filters dq_<name>_flagged__<column> and dq_<name>_score__<column>. The stem of average_line_length_filter is avg_line_length_filter, and the score column of mojibake_filter is dq_mojibake_filter_regex_score__<column>.
Keyword topics None. Report entries only.
Column statistics None. One result for each column named, no setting. percentiles reports the 5th, 25th, 50th, 75th, and 95th percentiles.
Label balance None. No setting.
Language dq_lang__<column>, dq_lang_score__<column>, and dq_language_detection_flagged__<column>. You need a View on every format.
PII detection dq_pii_status__<column> (pii_found or clean), dq_pii_types__<column> (the kinds of PII found), and dq_pii_flagged__<column>. A sample of the rows is read.
Taxonomy dq_taxonomy_<framing>_l1__<column>, _l2, _l3, and _margin for each framing. The HTML report is listed above.
Deduplication dq_<name>_row_id__<column>, dq_<name>_flagged__<column>, dq_<name>_reason__<column>, and dq_<name>_representative_row_id__<column>. The reason is exact_duplicate or near_duplicate on a flagged row, representative on the first row of a group, and unique on every other row.
Data quality specification evaluation dq_spec_eval_pass (true when the row passed every rule, false when it failed one, null when the row was not judged), dq_spec_eval_failed_criteria (the ids of the rules the row failed), dq_spec_eval_errored_criteria (the ids of the criteria that could not be evaluated), and dq_spec_eval_criteria (one entry per criterion, keyed by its id, with the verdict or the value, the reason, a confidence, and source, which is code, llm, prompt, or context). The rows outside a DownSampleRecordCount sample carry null in all four.
Exploratory analysis None. The findings are in aggregate/exploratory/findings.json.

What the example job wrote

The following excerpts come from the job of the Use example. The aggregate report opens with total_volume and total_samples, the number of rows read, and derived_inputs when an operation ran without a View. Each measurement's key holds one entry per View, with view, the columns read, and result.

On a job without a View, the column read is a derived column, so the row-level report names its columns in the form dq_word_count__dq_input__completion and carries the derived text in dq_input__prompt and dq_input__completion. The following row of the example job passed every criterion, shown with two of its five criterion entries, one decided by code and one by a model:

{ "ticket_id": "T10005", "prompt": "Can I share the annual plan with a colleague?", "completion": "Yes. Go to Members, choose Invite, and enter their email. They get an invitation that stays valid for seven days.", "category": "feature", "dq_input__prompt": "Can I share the annual plan with a colleague?", "dq_input__completion": "Yes. Go to Members, choose Invite, and enter their email. They get an invitation that stays valid for seven days.", "dq_word_count__dq_input__prompt": 9, "dq_word_count__dq_input__completion": 20, ... "dq_spec_eval_pass": true, "dq_spec_eval_failed_criteria": [], "dq_spec_eval_errored_criteria": [], "dq_spec_eval_criteria": { "answers_the_question": { "verdict": true, "value": null, "reason": "The completion directly explains how to share the plan by inviting a colleague, which answers the question asked.", "confidence": 0.9, "source": "llm", "error_type": null }, "completion_min_length": { "verdict": true, "value": null, "reason": "completion has 20 word(s), threshold is 8", "confidence": 1.0, "source": "code", "error_type": null }, ... } }

Reading the results for a decision

  • The spec_eval entry of aggregate/quality_report.json gives the counts of rows judged, passed, and failed, and the pass_rate and top_failure_modes of each criterion tell you which criterion the dataset fails most often and why. The criteria over the whole dataset are under aggregate_criteria, each with its value and its verdict.

  • The entries of the built-in metrics tell you whether the rows have the length and the variety that you expect.

  • The exploratory findings name problems that your data quality specification does not cover. A finding that matters becomes a criterion in your specification or a filter in data transformation.

  • The rows to act on are in record_level/evaluated_data.jsonl, filtered on the dq_ columns, such as every row where dq_spec_eval_pass is false.

Limits and quotas

The following limits apply to data quality evaluation.

Limit Value
Concurrent DQ jobs per account in each Region 3. Every job of the account counts, from any entry point. When three jobs run, a new CreateJob call fails with ResourceLimitExceeded. You can request an increase to the Maximum number of concurrent data quality evaluation jobs quota in the Service Quotas console.
Datasets per job 1
Input dataset size 18 GB in total for a job that runs exploratory_analysis, 30 GB for a job that runs spec_eval, and 90 GB for every other job. A job with several operations takes the smallest of their limits. A larger dataset fails the job.
Operations per job 1 to 50. default_metrics, spec_eval, data-validation, and exploratory_analysis at most once each.
Rows that spec_eval judges Every row, or DownSampleRecordCount rows, a positive integer.
Context documents for spec_eval At most 1,000 files, each at most 10 MiB and 200 MiB in total, with the extensions .txt, .md, .html, .htm, .json, .jsonl, .ndjson, .csv, .tsv, .pdf, and .docx.
Rows for taxonomy At least 10, none longer than 49,000 characters.
Job duration 24 hours
Evaluation entries per dataset The newest 100
Regions See Regions and limits for SageMaker AI Data Preparation.

See Regions and limits for SageMaker AI Data Preparation for the Regions and the quotas that apply to every capability. For SageMaker AI quotas, see SageMaker AI endpoints and quotas in the AWS General Reference.

Troubleshooting

A problem appears in one of three places:

  • The CreateJob call refuses the request, and no job exists.

  • You create the job, and it later reports the JobStatus Failed with a FailureReason.

  • A call on an existing job is refused.

A malformed request returns ValidationException with a text that names the field at fault and what the service accepts there. Fix the request as the text says and submit it again. A ThrottlingException with the text Rate exceeded means that too many CreateJob calls arrived from one account in a short time, so retry with exponential backoff and submit jobs one at a time. An AccessDeniedException that names sagemaker:ImportHubContent means that the caller lacks the hub permissions that Permissions and billing lists for the identity that submits the job. A DescribeJob or StopJob call within a few seconds of CreateJob can return ResourceNotFound before the job becomes readable, so retry it for a few seconds. The Amazon SageMaker Python SDK raises the same texts, and with wait=True it raises DataPreparerJobFailedError with the text Job <arn> failed: <FailureReason> when the job fails. The following entries cover the cases whose text does not say what to do, and the files that a failed job writes.

ClientError during Validating or Preparing

The job fails after Validating or Preparing with a FailureReason that starts with ClientError: and says what is wrong and what to do. A job that fails here writes nothing to its output folder. Two causes need more than the text gives:

  • The service could not assume the execution role because its trust policy does not allow sagemaker.amazonaws.com to assume it, which a role that trusts only job.sagemaker.amazonaws.com fails on.

  • The dataset is larger than the size limit for the job's evaluations.

To resolve the failure:

  1. Fix the cause that the text names.

  2. Submit the job again.

The dataset has an older document schema

The job reports Failed after it starts, with a FailureReason that names the dataset ARN, the dataset's document schema version, and the version that the job requires. To resolve the failure:

  1. Register the data as a dataset with DataSet.create, which writes a document that meets the requirement.

  2. Submit the job again on that dataset.

ClientError that points to CloudWatch Logs

The job reports Failed. The following FailureReason message shows this error:

ClientError: The job failed due to invalid input or configuration. Verify your request parameters and input data.. Check CloudWatch Logs: log-group=/aws/sagemaker/Job/DataQualityEvaluation/, stream=job-name/evaluation-created-at

The text is the same for every input that the job could not read or evaluate. To resolve the failure:

  1. Find the cause. For a spec_eval job that was given context documents, open spec_eval/context_documents.json in the output folder, whose failure entry names the document at fault and what to change. For every other cause, open the CloudWatch Logs stream that the text names.

  2. Fix the cause. The following table lists the causes recorded in the log, each with its fix.

  3. Submit the job again.

Cause Fix
A dataset without default columns (Nova_RLVR, or Custom without prompt and completion columns) was evaluated without a View. Add a View to every operation.
A line of the file is not a JSON object. Repair the line.
A SQLite file holds more than one table. Keep one.
The file's extension is not one that the job reads. Rename the file with the extension of its format.
The data quality specification produced no verdict for any row. Revise it.

If an operator fails, the job ends, with the following outcomes:

  • The results of the operations that finished before that operator stay in the output folder.

  • The job writes no entry to the dataset.

A failed job might have no metadata.json, so list the output location to find its files.

The job fails after it wrote every output file

The output files are all in the output folder, but the job could not add its entry to the dataset's registry entry, and with wait=True the SDK logs Compute succeeded but Hub finalization did not. To resolve the failure:

  1. Give the execution role sagemaker:ImportHubContent and sagemaker:UpdateHubContent permissions on the dataset's hub content and on its hub.

  2. Submit the job again.

The failed job has no MetadataS3Uri, so list the S3OutputPath prefix for a key that contains the job name to find its files.

Where the job's log is

The job's log lines are in your account in CloudWatch Logs, in the log group /aws/sagemaker/Job/DataQualityEvaluation/ and the stream job-name/evaluation-created-at. The DescribeJob response gives both names in the status message of the Executing transition, and every FailureReason written after that point repeats them. With evaluate(..., logs=True), the SDK prints the stream while the job runs.

Where you go next

You pass the same dataset ARN to the next step, because the job never creates a new dataset or a new version. The results lead to one of two steps:

  • Data transformation reads the dq_ columns of the row-level report and drops, filters, or modifies rows on them, such as the rows where dq_spec_eval_pass is false or dq_pii_status__<column> is pii_found.

  • Synthetic data generation is the next step when you generated the dataset: update the generation specification with what the findings showed and generate a new version of the dataset.

When you evaluate again, pass the job's spec_eval/ folder as CodeArtifactS3Uri to the next job on the new dataset or version, and the same checks run again.

The following pages of this guide describe the steps and the concepts around data quality evaluation:

The following references document the SDK and the API:

  • The Amazon SageMaker Python SDK reference for the data preparation module, sagemaker.ai_registry, documents DataPreparer, DatasetJobConfig, and JobCategory.

  • The Amazon SageMaker AI API Reference documents CreateJob, DescribeJob, ListJobs, StopJob, and DeleteJob, and the schema discovery operations ListJobSchemaVersions and DescribeJobSchemaVersion.

The following guides explain the tools that this page uses: