Data quality evaluation
What it is
With data quality evaluation, you measure the quality of a text or tabular dataset before you use it to train or evaluate a model. You run a job on a dataset that is registered in your hub and choose the evaluations in the job's evaluation configuration. SageMaker AI reports the results for the whole dataset and for each row. The job reads the dataset and never modifies it.
The job writes an aggregate report for the whole dataset and a row-level report, a copy of your rows with results that the job adds as columns. Some evaluations write more files, such as the findings file of the exploratory analysis.
You can evaluate dataset quality in three ways, and you can combine them in one job:
-
Built-in metrics: metrics that the service provides and that you choose by name, such as
word_countorpii_detection, for length and format, duplicates, personally identifiable information (PII), language, topics, and diversity. -
Data quality rules: Boolean conditions that each row must pass. A rule can check the fields of each row or the dataset as a whole. You write your rules in a data quality specification file, which can also ask for a label or a score for each row. A large language model (LLM) can judge a rule with a prompt that you write, a custom LLM-as-a-judge criterion, on its own or against documents that you provide.
-
Exploratory analysis: an agent that reads the dataset and writes the problems it finds, each with a title and the rows that show it, to a findings file, for a dataset where you do not yet know what to check. To act on the findings, see Reading the results for a decision.
Only data quality rules use a specification file. You choose built-in metrics and exploratory analysis by name in the job's evaluation configuration. Data quality rules are the recommended way, because they check your rows against conditions that you define and name the rules that each row failed.
When to use it
Use data quality evaluation when:
-
You want to check the quality of a dataset that you generated with synthetic data generation (SDG) before you use it for training or customization.
-
You want to bring your own dataset (BYOD) for training and have its format validated and its quality measured first.
-
You want every row checked against data quality rules that you have written.
-
You want to check for duplicate rows.
-
You want the dataset explored for problems, because you do not yet know what to check.
-
You want to know which topics the dataset covers.
How it works
A data quality evaluation is a data preparation job with the job category
DataQualityEvaluation, created with the same CreateJob
operation as the other data preparation jobs.
The job
A job has five parts:
-
The job name, a name that no existing job has.
-
The execution role, the IAM role that the job uses to read the dataset and write its files.
-
The input dataset, one registered dataset.
-
The output location, an Amazon S3 location in the job's AWS Region.
-
The evaluation configuration, the list of operations to run, each an operator and its settings.
The dataset
A job evaluates one dataset, named by its ARN in InputDataConfig. The
dataset is hub content of type DataSet in a hub in your account. Its
registry entry names the file format in DatasetFileFormat, which
decides the columns that the operators read when you name
none.
The job lifecycle
CreateJob returns the job ARN. DescribeJob reports a
JobStatus and a SecondaryStatus with a message for each
status the job passed through. The JobStatus is
InProgress, Stopping, Completed,
Failed, or Stopped. The secondary statuses of a successful
job, in order, are the following.
| Secondary status | What the job is doing |
|---|---|
Validating |
Validates the Region, the Amazon S3 locations, and the data quality specification or code artifact named for the job. |
Preparing |
Prepares the evaluation artifacts. |
Executing |
Executes the operations. |
Publishing |
Writes the output files to the output location. |
Finalizing |
Adds the job's entry to the dataset's registry entry. |
Completed |
Done, with MetadataS3Uri, the location of the job's
metadata.json, inside the returned configuration of the
DescribeJob response. |
A Failed job has a FailureReason. A job that has reached
Publishing can no longer be stopped. A stopped job has no
FailureReason, writes nothing under its own folder, and adds no entry
to the dataset.
The output location
Every job writes to its own folder in the output location, so several jobs can
share one S3OutputPath. The job also records its evaluation on the
dataset: it adds an entry that points to its output to the JobMetadata
list of the dataset's entry in your hub. It creates no new dataset and no new
version, so the dataset's ARN and version do not
change.
The operations
EvaluationConfig.Operations lists the operations of the job. Each
operation names an Operator, a built-in metric or an evaluation from
the table under
Operators,
and an optional Config with its settings. The following operation
counts the words of the prompt column. In the following example,
replace prompt with the name of your text
column:
{"Operator": "word_count", "Config": {"View": [{"Role": "Text", "Column": "prompt"}]}}
Config.View names the columns that the operator reads, one entry for
each column, each a role and a column name. The roles are the
following:
-
Text, a column of text, read by the text metrics. -
Column, a numeric or categorical column, read by the column statistics, with one result for each column. -
Label, a label column, read by the balance metrics. -
Messages, a column of conversation turns.
The other keys of Config are the operator's settings, and the service
refuses a key that the operator does not define when you submit the job. The
following operation detects the language of the prompt column and flags
the rows that are not English, with the accepted languages and the lowest accepted
score as
settings:
{"Operator": "language_detection", "Config": {"View": [{"Role": "Text", "Column": "prompt"}], "languages": ["en"], "min_score": 0.5}}
Without a View, an operator reads the default text columns of the
dataset's format: the prompt column and, where the format has one, the completion
column. You can run the same operator on two columns as two operations in one job.
The service refuses the same operator with the same Config twice as a
duplicate.
Operators
An operator is a measurement, a check, or an analysis that the job runs on your
dataset. Every built-in metric writes its summary to the aggregate report, and some
also write a column for each row to the row-level report. The following table lists
each operator group, what it does, the columns it reads, and its operators: 41
built-in metrics and the four evaluations default_metrics,
spec_eval, exploratory_analysis, and
data-validation.
| Operator group | What it does | Columns read | Operator |
|---|---|---|---|
| Data quality rules | Evaluation of every row, or a sample, and the whole dataset against the rules in your data quality specification file. | The whole row. | spec_eval |
| Exploratory analysis | An agent that explores the dataset and writes its findings. | The whole row. | exploratory_analysis |
| Text quality | The length, format, and variety of the text.
effectiveness: one score for the size of the dataset, no
column
read. |
One text column, role
Text. |
word_count, character_count,
compression_ratio, format_distribution,
plain_text_ratio, content_structures,
uniqueness,
ngram_diversity, mif_score,
effectiveness |
| Text filters | Flags on the rows to filter out, with no row removed: text that is mostly URLs, text with few letters and digits, repeated words or characters, broken character encoding, text that is too short or too long, and lines that are too short. | One text column, role
Text. |
urls_filter, alphanumeric_filter,
word_repetition_filter,
char_repetition_filter, mojibake_filter,
character_length_filter,
average_line_length_filter |
| Keyword topics | The rows for each topic from a keyword list that you provide, and the share of your topics that the dataset covers. | One text column, role
Text. |
keyword_topic_counts,
keyword_topic_coverage |
| Column statistics | A statistic for each numeric column, and the counts of null values, non-numeric values, the most frequent value, and the frequency of each value. | One or more columns, role
Column. |
min, max, mean,
median, std_dev, skewness,
kurtosis, percentiles,
count_zero, count_null,
count_non_numeric, mode,
value_frequency |
| Label balance | How evenly the label values are distributed, as a balance score and as a ratio, over all labels and for each label column. | One or more columns, role
Label. |
data_balance, data_balance_ratio,
field_balance, field_balance_ratio |
| Language | The language of each row, and a flag on the rows whose language is not in your list. | One text column, role Text, required on every
format. |
language_detection |
| PII | Flags on the sampled rows that contain personally identifiable information (PII). | One column, role Text or Messages,
required on every
format. |
pii_detection |
| Duplicates | Flags on the rows whose text repeats another row's, byte for byte or by similarity, with no row removed. | One text column, role
Text. |
exact_dedup, fuzzy_dedup |
| Taxonomy | A three-level taxonomy of the rows for each framing that you
request, task, domain, or
quality. |
One text column, role
Text. |
taxonomy |
| Default metrics | Nine of the text quality metrics at once on the format's default text columns. | The format's default text columns. | default_metrics |
| Format validation | Every record checked against the file format that you declared when you registered the dataset. | The whole record. | data-validation |
Metric settings
The following built-in metrics take settings in Config, beside the
View.
| Operator | Setting | Values |
|---|---|---|
uniqueness |
CaseSensitive |
true by default. |
ngram_diversity |
N |
1 to 8, with 1 the default. |
mif_score |
mode |
corpus or per_row. |
character_length_filter |
min_length, max_length |
The shortest and the longest accepted text, in characters. |
keyword_topic_counts,
keyword_topic_coverage |
topic_keywords |
A map from each topic to its keywords. |
keyword_topic_counts,
keyword_topic_coverage |
default_topic |
The topic for the rows that no keyword matches. |
language_detection |
languages |
A list of language codes. |
language_detection |
min_score |
The lowest accepted score for the detected language, with the rows below it flagged. |
taxonomy |
framings |
A list from task, domain, and
quality, with all three the
default. |
exact_dedup |
Normalize |
false by default. |
fuzzy_dedup |
JaccardThreshold |
The similarity at which two rows count as duplicates, with 0.7 the default. |
fuzzy_dedup |
Lowercase |
true by default. |
Data quality specification evaluation
spec_eval evaluates each row, or a sample of the rows, and the whole
dataset against the data quality specification file named in its Config. It reads
whole rows and takes no View. Its Config takes the
following
keys.
| Key | What it is | Required |
|---|---|---|
SpecS3Uri |
The Amazon S3 location of your data quality specification file, a .md
or .txt file in the prose or the structured
form. |
Exactly one of SpecS3Uri and
CodeArtifactS3Uri. |
CodeArtifactS3Uri |
The spec_eval/ folder that an earlier job wrote, as
it is or with a judge_code.py that you modified. The job
runs that code and reads no specification
file. |
Exactly one of SpecS3Uri and
CodeArtifactS3Uri. |
ContextDocumentsS3Uri |
An Amazon S3 folder of context documents that your criteria are judged against. The folder also adds the service's own faithfulness check. | Optional. |
DownSampleRecordCount |
The number of rows to judge instead of all of them. The rows
outside the sample carry null in the four specification
columns of the row-level
report. |
Optional. |
ModelConfig.JudgeModelConfig.ModelId |
The model that judges the rows,
us.anthropic.claude-sonnet-5 by default. The accepted
ids are us.anthropic.claude-sonnet-5,
us.anthropic.claude-sonnet-4-6,
us.anthropic.claude-opus-4-6-v1,
us.anthropic.claude-sonnet-4-5-20250929-v1:0,
us.anthropic.claude-sonnet-4-20250514-v1:0,
us.anthropic.claude-haiku-4-5-20251001-v1:0, and
openai.gpt-oss-120b-1:0, and the service refuses another
id. |
Optional. |
Exploratory analysis
exploratory_analysis runs an agent that reads the dataset and writes
a list of findings, each with a title and the rows that show it, to the findings
file aggregate/exploratory/findings.json. It takes no
Config and writes no row
column.
PII detection
pii_detection uses Amazon Bedrock models to find personally identifiable information
(PII). It reads a sample of the rows of one Text or
Messages column, so its View has exactly one entry and is
required on every format. It flags each sampled row that contains PII and records the kinds found. It
takes no other
setting and never writes a detected value to the
output.
PII detection finds direct identifiers only: information that identifies an individual on its own, such as a name or an account number. It does not find quasi-identifiers: information that identifies an individual only in combination with other information.
The categories it detects are the following:
-
Names and contact details: private names, public names, email addresses, and phone numbers.
-
Addresses and locations: full addresses, partial addresses, and GPS coordinates.
-
Government and financial numbers: card numbers, card codes and expiry dates, bank account numbers, PINs, Social Security numbers, driver's license numbers, passport numbers, and other government IDs.
-
Account, device, and network identifiers: IP addresses, MAC addresses, user names, device and tracking IDs, and account and membership numbers.
-
Vehicle identifiers: license plates and vehicle identification numbers.
-
Passwords and credentials: passwords and other credentials.
-
Dates and age: personal dates, such as a date of birth, and age.
-
Web addresses: public URLs and private URLs.
Taxonomy
taxonomy groups the rows of one text column into a three-level
taxonomy for each framing that you request, task, domain,
or quality. It writes the three levels of each framing to the row-level
report and an HTML report.
Deduplication
exact_dedup and fuzzy_dedup flag the duplicates in one
text column and remove no row. Each flagged row carries a reason and the id of the
first row of its group.
Default metrics and format validation
default_metrics runs word_count,
character_count, format_distribution,
plain_text_ratio, compression_ratio,
uniqueness, ngram_diversity, mif_score,
and effectiveness on the format's default text
columns and takes no Config. On a format without default columns, list
the metrics yourself with a
View.
data-validation checks every record against the format that you
declared, and it reads JSON Lines and Parquet datasets. It takes no
Config and must be the only operation in its
job.
How a data quality specification works
A data quality specification file states your data quality rules, the conditions that each row or the whole dataset must pass, and any measurements that you want taken. You write the rules in your own words, such as that a reply must answer the question or that a completion must have at least eight words. The job turns the file into criteria, runs them on every row or on a sample, and reports which rows failed which rule and the value of each measurement.
The two forms
You write a data quality specification in one of two forms:
-
The prose form: rules written as sentences, with no special syntax, for a first specification or for rules that are still changing.
-
The structured form: a
## Criteriasection whose entries name each criterion and state how it is decided, for an evaluation that you repeat with stable criterion ids.
The ## Criteria heading is what makes a file structured, and both
forms are read the same way whether the file name ends in .md or
.txt.
Rules and measurements
Each rule or measurement that you write becomes a criterion, the unit that the job evaluates. A criterion is a rule, with a pass or fail verdict, or a measurement, with a value and no verdict, for every row or once for the whole dataset. From a prose file, a model derives one criterion for each rule or measurement and takes a threshold from the sentence when it states one. From a structured file, the job reads the ids and the descriptions as written.
How a criterion is decided
The job decides a criterion from your description, with your own prompt, or against your context documents, and a criterion can combine the last two. In the prose form the job chooses from your sentences, and in the structured form you choose with the keys of the entry:
-
From your description. The service turns the description into generated code or into a judge prompt for a model, whichever fits the criterion. This is the default.
-
With your own prompt. A
promptline sends your text to the model as written, in place of the judge prompt that the service would write. This is a custom LLM-as-a-judge criterion: the model judges each row with your prompt, as a rule or as a measurement. -
Against your context documents. The documents that you give the job are the reference for the criterion. In the structured form a criterion marked
context: documentsis judged this way, and in the prose form a rule or a measurement that refers to the documents is judged this way.
The following structured data quality specification shows one criterion decided from its description, one judged with your prompt, and one judged with your prompt against the documents:
Example Data quality specification using three evaluation methods
# Replies judged three ways ## Criteria - id: completion_present description: The completion field is present and is not blank or whitespace only. - id: reply_is_actionable prompt: You are reviewing a customer-support reply. Read the PROMPT (the customer's message) and the COMPLETION (the agent's reply). Answer PASS if the reply tells the customer exactly what will happen next or what they must do next. Answer FAIL if the reply only acknowledges the problem, restates it, or gives no next step. description: The reply tells the customer what happens next or what to do next. - id: reply_follows_guide context: documents prompt: You are reviewing a customer-support reply against the reference documents. Read the COMPLETION (the agent's reply) and the documents. Answer PASS if the reply follows every rule that the documents state for replies. Answer FAIL if it breaks one, and name the rule. description: The reply follows the rules in the style guide.
The service decides completion_present from its description,
judges reply_is_actionable with your prompt, and judges
reply_follows_guide with your prompt against the
documents.
Context documents
You can give the job context documents in an Amazon S3 folder, named by
ContextDocumentsS3Uri in the spec_eval operation. When
you do, the service adds its own check, builtin_faithfulness, which
judges whether each row is grounded in the documents: every factual claim in the
row must be stated by the documents or follow from them. A row that fails this
check counts as failed, so give the job documents that state the facts that your
rows rely on. The pass rate of a criterion judged against the documents, the
service's own or yours, is under context_criteria in the aggregate
report.
The data quality specification and the context documents control how your rows are judged. The job sends your specification, and your context documents when you give them, to the model as written, and the service does not detect or remove instructions in that text. The model also follows text that you did not write as a rule, such as instructions inside a document that you downloaded, and such text can change the verdicts. Before you submit a job, review the specification and the documents, and use only files that you wrote or that you trust.
What the job writes
A spec_eval operation writes the following to the output folder,
the same for both forms:
-
The criteria that the job derived or read, in
spec_eval/extracted_criteria.jsonin the output folder, so that you can check how it read your rules. -
A
spec_evalentry in the aggregate report, with the counts and the pass rates over the dataset. -
Four columns for each row in the row-level report: the verdict, the rules that the row failed, the criteria that could not be evaluated, and each criterion's result.
Data quality specification examples
Structured data quality specification example
The following structured data quality specification was run on the 200-row customer-support
dataset of the Use example. It
holds a rule decided by code, a rule decided by a model, a label, a score, and a
rule over the whole dataset, five of the file's seven criteria, with the
description of tone shortened. The table at the end of this section
explains each
key.
# Data quality rules for the customer-support training dataset ## Stratify by - category ## Criteria - id: completion_min_length type: code! threshold: {"min_words": 8} description: The completion contains at least min_words words, counted by splitting on whitespace. - id: answers_the_question type: llm! description: The completion answers what the customer asked in the prompt. A reply that restates the question, changes the subject, or refuses without a reason fails. - id: tone kind: metric values: [formal, friendly, curt] description: Classify the tone of the completion as formal, friendly, or curt. - id: helpfulness kind: metric values: 0 to 1 description: Score how completely the completion resolves the customer's problem, from 0.0 (does not help) to 1.0 (fully resolves it), in steps of 0.25. 0.5 means a partial answer that leaves the customer with a next step to find on their own. - id: answer_rate scope: aggregate inputs: [answers_the_question] threshold: {"min_rate": 0.9} description: At least min_rate of the rows pass answers_the_question.
The job judged all 200 rows, of which 173 passed and 27 failed.
completion_min_length had a pass rate of 0.96 and
answers_the_question of 0.87, with a pass rate for each value of
category, such as 0.84 on the billing rows.
answer_rate returned one verdict, a fail, with the reason
174/200 rows passed answers_the_question (rate=0.870), threshold=0.9.
The labels of tone and the scores of helpfulness
appear per row as the value of their entries, with no pass rate and
no distribution in the aggregate report.
Sections and keys of a structured data quality specification
The following table lists the sections and the keys of a structured data quality specification.
| Section or key | What it is |
|---|---|
## Stratify by |
An optional section before ## Criteria with one
column name on each line. Every row rule then reports its pass
rate for each value of that
column. |
## Criteria |
The section that holds the criteria, one entry each, and the heading that makes the file structured. |
id |
The name of the criterion, which starts its entry and keys its result in the report and in every row. |
kind |
rule, the default, for a pass or fail verdict, or
metric for a measurement with a value and no
verdict. |
type |
What decides the criterion: auto, the default,
lets the job choose, code and llm
request generated code or a model, and code! and
llm! lock that choice, which the report records.
On a criterion with a prompt or a
context: documents line, leave the type
line out, because the model always judges such a
criterion. |
scope |
row, the default, for a result per row, or
aggregate for one result over the whole
dataset. |
values |
The results that a measurement may return, as labels in
brackets such as [formal, friendly, curt] or as a
range such as 0 to 1. |
threshold |
A JSON object with the numbers that a rule compares against,
such as {"min_words": 8}, on rules
only. |
inputs |
The row criteria that an aggregate criterion reads, such as the row rule whose pass rate it requires. |
reads |
The fields of your rows that the criterion uses, such as
[completion], echoed in the report. A field that
your rows do not have makes the criterion not evaluable on any
row. |
context: documents |
The model judges the criterion against the documents that you
provide. In the row-level report such a criterion has the
source context, its result in
outcome instead of verdict, such as
fail, measured,
not_applicable, or not_evaluable, and
the passage it relied on in quote and
evidence. |
prompt |
The exact text that the model judges with, one paragraph for
each line, sent as written in place of the judge prompt that the
service writes from the description. The
description stays for the report. With a
context: documents line as well, the model judges
with your prompt against the
documents. |
description |
What the criterion evaluates, in words, which the generated code or the model follows. |
Inputs
A job takes one registered dataset, often the dataset that
Synthetic data generation produced and
registered as hub content of type DataSet, whose arn you pass
to the job unchanged. You can also bring your own dataset (BYOD), register it in a hub,
and evaluate it the same way. For data quality rules, you also need the data quality specification
file and, if your criteria refer to documents, the context
documents.
In the snippets on this page, replace amzn-s3-demo-bucket
with the name of your Amazon S3 bucket.
The dataset file and its format
The job reads one dataset from one Amazon S3 location, a single file or a folder of Parquet shards with identical columns. The file's extension decides how it is read:
-
.jsonl: JSON Lines, one JSON object per line. -
.csv: CSV with a header row that names every column. -
.parquet: Parquet. -
.sqliteor.db: a SQLite database with one table.
A file with any other extension fails the job. The job reads text and tabular
data, not image, audio, or video data. The location must be in the job's Region and
in your account. Column names that start with dq_input__ are reserved
for the job's own derived columns. The dataset's files total at most 18 GB for a job
that runs exploratory_analysis, 30 GB for a job that runs
spec_eval, and 90 GB for every other job. A job with
several operations takes the smallest of their limits.
When you register the dataset, you name its format in
DatasetFileFormat, and most training formats give the job default text
columns to read when an operation has no View. The row shape of each
format is described in
Datasets and hub content.
Without a View, an operator reads the format's prompt column and, where
the format has one, its completion column, such as prompt and
completion for HF_Prompt_Completion.
Nova_RLVR, Nova_DPO, MTRL, and a
Custom dataset whose columns are not named prompt and
completion have no default columns, so every operator needs a
View on them. When your rows fit no named format, register the dataset
as Custom and name the column that each operator reads in a
View.
Registering your own data
To bring your own dataset (BYOD), upload the file to a prefix of its own in Amazon S3
and register that prefix as a dataset in a hub in your account. In the
Amazon SageMaker Python SDK, DataSet.create registers the dataset and returns the
DataSet whose arn you pass to the job. The registry entry
it writes has the DatasetType BYOD, the
DatasetFileFormat, and the data's
location:
{ "DatasetType": "BYOD", "DatasetFileFormat": "HF_Prompt_Completion", "DatasetS3Uri": "s3://amzn-s3-demo-bucket/datasets/support-conversations/" }
A dataset that DataSet.create registers has the document schema that
the job requires.
The data quality specification file
Upload your data quality specification file, with a name that ends in .md or
.txt, to Amazon S3. To judge rows against your own documents, upload them to
a folder of their own. You name both locations in the spec_eval
operation, with SpecS3Uri and ContextDocumentsS3Uri. If
the dataset came from synthetic data generation, you can pass the specification that generated it as
SpecS3Uri, because a generation specification is also a valid
specification for this job, which reads only the file that you name. Review the
specification and the documents before you upload them, as the Important note under
How a data quality specification works
says.
Use data quality evaluation
You can run data quality evaluation in the Amazon SageMaker Python SDK directly, or through a coding assistant with the core AI and ML skill of the Agent Toolkit for AWS. Both run the same job and write the same result files.
Permissions and billing
A data quality evaluation job needs the following permissions, whichever entry point submits it.
| Who | Permissions |
|---|---|
| The execution role | bedrock:InvokeModel and
bedrock:InvokeModelWithResponseStream,
s3:GetBucketLocation, read and write access to the
input and output locations, and
sagemaker:ImportHubContent on the hub that holds the
dataset. |
| The identity that manages jobs | sagemaker:DescribeJob and
sagemaker:StopJob on the job ARN,
sagemaker:DeleteJob, and
sagemaker:ListJobs. |
| The identity that submits the job | sagemaker:CreateJob on the job ARN,
iam:PassRole for the execution role to
job.sagemaker.amazonaws.com, and
sagemaker:ImportHubContent and
sagemaker:UpdateHubContent on the dataset and its hub,
so that the job can record its result on the
dataset. |
| The trust policy of the execution role | sagemaker.amazonaws.com allowed to assume the role.
This is a different service name from the one in the
iam:PassRole condition above, and the job needs both as
written. |
The job calls Amazon Bedrock models on your behalf, so the execution role needs permission to call them, and the models' pricing and on-demand quotas apply to the job. The following evaluations call a model:
-
The data quality specification evaluation calls a model once to derive the criteria from your specification. Then, for every row, it calls a model on each criterion that a model decides. Criteria that generated code decides cost no model calls. To try a specification on a sample first, set
DownSampleRecordCount. -
The exploratory analysis, PII detection, and taxonomy call a model on the rows they read.
The file token_usage_report.json in the output folder records the
models the job called and the tokens it used.
Results
A completed job writes the following files under its own folder in the output
location,
.
The following table lists each file, what it holds, and when the job writes
it.S3OutputPathdata-quality-evaluation/job-name/created-at/
| File | What it holds | When the job writes it |
|---|---|---|
The index file metadata.json |
The starting point of a run: the job, the dataset, the evaluation
configuration you sent, the time, and under Reports the
locations of the aggregate report, the row-level report, and the findings
and data quality specification evaluation files when the job writes them. Its own
location is MetadataS3Uri on the dataset's entry and in the
DescribeJob response. It does not list
token_usage_report.json or the taxonomy
report. |
Every completed job. |
The aggregate report aggregate/quality_report.json |
One entry per measurement over the whole dataset. | Every completed job except a data-validation job, which
writes its own report under
validation/. |
The row-level report
record_level/evaluated_data.jsonl |
A copy of your rows, unchanged, with the metric values and the rule
verdicts added as columns whose names start with
dq_. |
Every completed job, unless its only operation is
exploratory_analysis or it is a data-validation
job. |
The usage file token_usage_report.json |
The models the job called and the tokens it used, in total and for each model. | Every completed job. |
The folder spec_eval/ |
extracted_criteria.json, the criteria the job derived or
read. judge_code.py, the program that judged the rows.
judge_config.json, the model id, the token cap, and the
Region of the judging. reproduce.py, a script that replays
the judging on your own machine. judge_texts.json, for a job
that judged against your
documents. |
A job that ran
spec_eval. |
The findings file
aggregate/exploratory/findings.json |
The problems the analysis found, each with a title and
evidence, the counts and example rows it rests on. The same
list is under the exploratory_analysis key of the aggregate
report. |
A job that ran
exploratory_analysis. |
The taxonomy report
aggregate/taxonomy/ |
The taxonomy of the rows, where run-id is
the run identifier the job assigns. Its location is
taxonomy_report_path in the taxonomy entry of
the aggregate report. |
A job that ran taxonomy. |
The files under validation/ |
The validation set's metadata file, a report with the counts of valid
and failed records, the failure rate, and the record shape the job
detected, one line for each record that failed, and the indices of
those records, beside the job's own metadata.json and
token_usage_report.json. |
A data-validation
job. |
The columns that the operators write
The following table lists the columns that each operator writes to the row-level
report, where <column> stands for the column that the operator
read.
| Metrics | Row columns | Notes |
|---|---|---|
Text quality: word_count,
character_count, compression_ratio,
content_structures, and
mif_score in its per_row mode |
One column for each metric, named after the column read, such as
dq_word_count__prompt. |
The other five text quality metrics write report entries only. |
| Text filters | dq_<name>_flagged__<column> and
dq_<name>_score__<column>. |
The stem of average_line_length_filter is
avg_line_length_filter, and the score column of
mojibake_filter is
dq_mojibake_filter_regex_score__<column>. |
| Keyword topics | None. | Report entries only. |
| Column statistics | None. | One result for each column named, no setting.
percentiles reports the 5th, 25th, 50th, 75th, and 95th
percentiles. |
| Label balance | None. | No setting. |
| Language | dq_lang__<column>,
dq_lang_score__<column>, and
dq_language_detection_flagged__<column>. |
You need a View on every
format. |
| PII detection | dq_pii_status__<column> (pii_found or
clean), dq_pii_types__<column> (the
kinds of PII found), and
dq_pii_flagged__<column>. |
A sample of the rows is read. |
| Taxonomy | dq_taxonomy_<framing>_l1__<column>,
_l2, _l3, and _margin for each
framing. |
The HTML report is listed above. |
| Deduplication | dq_<name>_row_id__<column>,
dq_<name>_flagged__<column>,
dq_<name>_reason__<column>, and
dq_<name>_representative_row_id__<column>. |
The reason is exact_duplicate or
near_duplicate on a flagged row,
representative on the first row of a group, and
unique on every other
row. |
| Data quality specification evaluation | dq_spec_eval_pass (true when the row
passed every rule, false when it failed one,
null when the row was not judged),
dq_spec_eval_failed_criteria (the ids of the rules the
row failed), dq_spec_eval_errored_criteria (the ids of
the criteria that could not be evaluated), and
dq_spec_eval_criteria (one entry per criterion, keyed by
its id, with the verdict or the value, the reason, a confidence, and
source, which is code, llm,
prompt, or
context). |
The rows outside a DownSampleRecordCount sample carry
null in all four. |
| Exploratory analysis | None. | The findings are in
aggregate/exploratory/findings.json. |
What the example job wrote
The following excerpts come from the job of the Use example. The aggregate report
opens with total_volume and total_samples, the number of
rows read, and derived_inputs when an operation ran without a
View. Each measurement's key holds one entry per View,
with view, the columns read, and
result.
On a job without a View, the column read is a derived column, so the
row-level report names its columns in the form
dq_word_count__dq_input__completion and carries the derived text in
dq_input__prompt and dq_input__completion. The following
row of the example job passed every criterion, shown with
two of its five criterion entries, one decided by code and one by a
model:
{ "ticket_id": "T10005", "prompt": "Can I share the annual plan with a colleague?", "completion": "Yes. Go to Members, choose Invite, and enter their email. They get an invitation that stays valid for seven days.", "category": "feature", "dq_input__prompt": "Can I share the annual plan with a colleague?", "dq_input__completion": "Yes. Go to Members, choose Invite, and enter their email. They get an invitation that stays valid for seven days.", "dq_word_count__dq_input__prompt": 9, "dq_word_count__dq_input__completion": 20, ... "dq_spec_eval_pass": true, "dq_spec_eval_failed_criteria": [], "dq_spec_eval_errored_criteria": [], "dq_spec_eval_criteria": { "answers_the_question": { "verdict": true, "value": null, "reason": "The completion directly explains how to share the plan by inviting a colleague, which answers the question asked.", "confidence": 0.9, "source": "llm", "error_type": null }, "completion_min_length": { "verdict": true, "value": null, "reason": "completion has 20 word(s), threshold is 8", "confidence": 1.0, "source": "code", "error_type": null }, ... } }
Reading the results for a decision
-
The
spec_evalentry ofaggregate/quality_report.jsongives the counts of rows judged, passed, and failed, and thepass_rateandtop_failure_modesof each criterion tell you which criterion the dataset fails most often and why. The criteria over the whole dataset are underaggregate_criteria, each with its value and its verdict. -
The entries of the built-in metrics tell you whether the rows have the length and the variety that you expect.
-
The exploratory findings name problems that your data quality specification does not cover. A finding that matters becomes a criterion in your specification or a filter in data transformation.
-
The rows to act on are in
record_level/evaluated_data.jsonl, filtered on thedq_columns, such as every row wheredq_spec_eval_passisfalse.
Limits and quotas
The following limits apply to data quality evaluation.
| Limit | Value |
|---|---|
| Concurrent DQ jobs per account in each Region | 3. Every job of the account counts, from any entry point. When three
jobs run, a new CreateJob call fails with
ResourceLimitExceeded. You can request an increase to
the Maximum number of concurrent data quality
evaluation jobs quota in the Service Quotas console |
| Datasets per job | 1 |
| Input dataset size | 18 GB in total for a job that runs exploratory_analysis, 30 GB for
a job that runs spec_eval, and 90 GB for every other
job. A job with several operations takes the smallest of their limits. A
larger dataset fails the job. |
| Operations per job | 1 to 50. default_metrics, spec_eval,
data-validation, and exploratory_analysis at
most once each. |
Rows that spec_eval judges |
Every row, or DownSampleRecordCount rows, a positive
integer. |
Context documents for spec_eval |
At most 1,000 files, each at most 10 MiB and 200 MiB in total, with
the extensions .txt, .md, .html,
.htm, .json, .jsonl,
.ndjson, .csv, .tsv,
.pdf, and .docx. |
Rows for taxonomy |
At least 10, none longer than 49,000 characters. |
| Job duration | 24 hours |
| Evaluation entries per dataset | The newest 100 |
| Regions | See Regions and limits for SageMaker AI Data Preparation. |
See Regions and limits for SageMaker AI Data Preparation for the Regions and the quotas that apply to every capability. For SageMaker AI quotas, see SageMaker AI endpoints and quotas in the AWS General Reference.
Troubleshooting
A problem appears in one of three places:
-
The
CreateJobcall refuses the request, and no job exists. -
You create the job, and it later reports the
JobStatusFailedwith aFailureReason. -
A call on an existing job is refused.
A malformed request returns ValidationException with a text that names
the field at fault and what the service accepts there. Fix the request as the text says
and submit it again. A ThrottlingException with the text
Rate exceeded means that too many CreateJob calls arrived
from one account in a short time, so retry with exponential backoff and submit jobs one
at a time. An AccessDeniedException that names
sagemaker:ImportHubContent means that the caller lacks the hub permissions
that
Permissions and billing
lists for the identity that submits the job. A DescribeJob or
StopJob call within a few seconds of CreateJob can return
ResourceNotFound before the job becomes readable, so retry it for a few
seconds. The Amazon SageMaker Python SDK raises the same texts, and with wait=True it
raises DataPreparerJobFailedError with the text
Job <arn> failed: <FailureReason> when the job fails. The
following entries cover the cases whose text does not say what to do, and the files
that a failed job
writes.
ClientError during Validating or Preparing
The job fails after Validating or Preparing with
a FailureReason that starts with ClientError: and
says what is wrong and what to do. A job that fails here writes nothing to
its output folder. Two causes need more than the text
gives:
-
The service could not assume the execution role because its trust policy does not allow
sagemaker.amazonaws.comto assume it, which a role that trusts onlyjob.sagemaker.amazonaws.comfails on. -
The dataset is larger than the size limit for the job's evaluations.
To resolve the failure:
-
Fix the cause that the text names.
-
Submit the job again.
The dataset has an older document schema
The job reports Failed after it starts, with a
FailureReason that names the dataset ARN, the dataset's
document schema version, and the version that the job requires. To resolve
the failure:
-
Register the data as a dataset with
DataSet.create, which writes a document that meets the requirement. -
Submit the job again on that dataset.
ClientError that points to CloudWatch Logs
The job reports Failed. The following
FailureReason message shows this
error:
ClientError: The job failed due to invalid input or configuration. Verify your request parameters and input data.. Check CloudWatch Logs: log-group=/aws/sagemaker/Job/DataQualityEvaluation/, stream=job-name/evaluation-created-at
The text is the same for every input that the job could not read or evaluate. To resolve the failure:
-
Find the cause. For a
spec_evaljob that was given context documents, openspec_eval/context_documents.jsonin the output folder, whosefailureentry names the document at fault and what to change. For every other cause, open the CloudWatch Logs stream that the text names. -
Fix the cause. The following table lists the causes recorded in the log, each with its fix.
-
Submit the job again.
| Cause | Fix |
|---|---|
A dataset without default columns
(Nova_RLVR, or Custom without
prompt and completion columns) was
evaluated without a
View. |
Add a View to every
operation. |
| A line of the file is not a JSON object. | Repair the line. |
| A SQLite file holds more than one table. | Keep one. |
| The file's extension is not one that the job reads. | Rename the file with the extension of its format. |
| The data quality specification produced no verdict for any row. | Revise it. |
If an operator fails, the job ends, with the following outcomes:
-
The results of the operations that finished before that operator stay in the output folder.
-
The job writes no entry to the dataset.
A failed job might have no metadata.json, so list the output
location to find its files.
The job fails after it wrote every output file
The output files are all in the output folder, but the job could not add
its entry to the dataset's registry entry, and with wait=True
the SDK logs Compute succeeded but Hub finalization did not. To
resolve the failure:
-
Give the execution role
sagemaker:ImportHubContentandsagemaker:UpdateHubContentpermissions on the dataset's hub content and on its hub. -
Submit the job again.
The failed job has no MetadataS3Uri, so list the
S3OutputPath prefix for a key that contains the job name to find
its files.
Where the job's log is
The job's log lines are in your account in CloudWatch Logs, in the log group
/aws/sagemaker/Job/DataQualityEvaluation/ and the stream
.
The job-name/evaluation-created-atDescribeJob response gives both names in the status message
of the Executing transition, and every
FailureReason written after that point repeats them. With
evaluate(..., logs=True), the SDK prints the stream while the
job runs.
Where you go next
You pass the same dataset ARN to the next step, because the job never creates a new dataset or a new version. The results lead to one of two steps:
-
Data transformation reads the
dq_columns of the row-level report and drops, filters, or modifies rows on them, such as the rows wheredq_spec_eval_passisfalseordq_pii_status__<column>ispii_found. -
Synthetic data generation is the next step when you generated the dataset: update the generation specification with what the findings showed and generate a new version of the dataset.
When you evaluate again, pass the job's spec_eval/ folder as
CodeArtifactS3Uri to the next job on the new dataset or version, and the
same checks run again.
Related information
The following pages of this guide describe the steps and the concepts around data quality evaluation:
-
Preparing Data with SageMaker AI Data Preparation introduces the data preparation jobs and the hub that holds your datasets.
-
Datasets and hub content explains the dataset ARN, the row shape of each dataset format, and the registry entry to which this job adds its result.
-
Get started with SageMaker AI Data Preparation and Set up and prerequisites describe the execution role, the default Amazon S3 bucket, and the permissions that the caller needs.
-
How a data quality specification works describes the data quality specification you write for a data quality evaluation job.
-
Synthetic data generation is the step that produces a dataset that you evaluate here.
-
Data transformation is the step that drops, filters, or modifies the rows that this job flagged.
-
Regions and limits for SageMaker AI Data Preparation lists the Regions and the quotas that apply to every capability.
The following references document the SDK and the API:
-
The Amazon SageMaker Python SDK reference for the data preparation module,
sagemaker.ai_registry, documentsDataPreparer,DatasetJobConfig, andJobCategory. -
The Amazon SageMaker AI API Reference documents
CreateJob,DescribeJob,ListJobs,StopJob, andDeleteJob, and the schema discovery operationsListJobSchemaVersionsandDescribeJobSchemaVersion.
The following guides explain the tools that this page uses:
-
The Agent Toolkit for AWS User Guide explains how to install the toolkit and its AI and ML skill, named under Use data quality evaluation.
-
The Amazon CloudWatch Logs User Guide explains how to read the job log.