Auto-labeling
What it is
With auto-labeling, you describe how to label your data in a labeling specification, and SageMaker AI labels every row of an unlabeled dataset with Amazon Bedrock models. For each row, the job also measures how confident it is in the label. When you give it rows that people already labeled, it measures the labels' accuracy and certifies the rows that it accepts. The accepted rows meet an accuracy target that you set, at 95% confidence. The job sends the rows that miss the bar to a separate review dataset for people to label.
Auto-labeling supports the following tasks:
| Task | Type | What each row gets |
|---|---|---|
| Classification | labeling.classification |
One label from the label set in your specification |
| Named entity recognition | labeling.ner |
Every entity span in the text, with its type |
| Extractive question answering | labeling.qa |
The answer, copied from the passage |
| Ordinal scoring | labeling.ordinal |
A score on the scale that your specification states |
| LLM-as-a-judge | labeling.llm_judge |
A score against the rubric in your specification |
| Summarization | labeling.summarization |
A summary of the text, graded by other models |
Use auto-labeling when:
-
You have unlabeled data, such as support tickets, reviews, or documents, and need labels to fine-tune or evaluate a model.
-
You can't afford to label every row by hand, but you need to know how accurate the labels are.
-
You want people to label only the rows that the models are unsure about.
How it works
An auto-labeling job is a synthetic data generation (SDG) job in labeling mode. Instead of
generating new records, it labels the rows of your input data. You start it the same way
as an SDG job, with generate() and a
generation_config, and you add dataset_options to say which
task to run. The job works with the following:
- Labeling specification
-
A Markdown (
.md) or text (.txt) file that describes the task: the use case, the labels and what each one means, and rules for the cases that are easy to confuse. The job sends the specification to the labeling models with each row, so it's the main way that you control label quality. For how to structure one, see Write a labeling specification. - Input data
-
The unlabeled rows, as a JSON Lines file in Amazon S3. Each row is a JSON object. You name the fields that the models read, such as
subjectandbody, intext_fields. Give each row anid. The job uses it to join gold rows, predictions, and review answers back to the row. - Evaluation data (optional)
-
Rows that people already labeled, called gold rows. A gold row either has the same
idas an input row, or carries every field intext_fields. In the second case, the job labels it on its own. The job labels these rows like any other row, and then compares its labels with the gold labels to measure accuracy. The models never see the gold labels. Without evaluation data, the job can't measure accuracy, so a corpus run labels at a fixed confidence bar and certifies nothing. - Labeling models
-
The Amazon Bedrock models that you list in
ModelConfig.SampleGenerationModelConfigs. How the job measures confidence depends on the task. The following table shows the confidence and the number of models for each task.Task Confidence Models Classification, named entity recognition, question answering The share of models that agree on the label 2 or more, so that the models can vote Ordinal scoring, LLM-as-a-judge How closely 5 samples from one model agree 1 Summarization No confidence for each row. Grader models score a sample of the summaries. 1 model labels each row, plus 1–3 graders The voting and resampling run only on a run with routing, called a corpus run. Without routing, the first model labels each row alone.
- Output datasets
-
Each completed job registers the labeled rows as hub content of type
DataSetin a hub in your account. A corpus run also registers a secondDataSetthat holds the rows to review. For more information about datasets, see Datasets and hub content.
How the job decides which labels to keep
On a corpus run with evaluation data and an accuracy target, the job does the following for the tasks that it can certify: classification, named entity recognition, and question answering.
-
Labels every row and scores each label. Each row gets a label and a confidence from 0 to 1.
-
Measures accuracy on the gold rows. The job compares its labels on the gold rows with the people's labels.
-
Picks the confidence cutoff. The job searches for the lowest cutoff whose accepted gold rows still meet your target at 95% confidence. This way, it accepts as many rows as possible. A row is accepted only if its confidence is at or above the cutoff and at least 75% of the models agree on its label.
-
Certifies or routes. If a cutoff meets the target, the job accepts the rows at or above it and certifies them. It also draws a random spot-check sample from the accepted rows for people to confirm. For example, at a 0.90 target it draws 29 rows. Every other row goes to the review dataset. If no cutoff meets the target, the job accepts nothing and routes every row.
A certificate needs at least 50 scored gold rows, and they must be a random sample of the same data as the input rows. A corpus run of a certifiable task that can't certify, because it has no evaluation data or no target, accepts rows at a fixed confidence bar of 0.85. With a target but no evaluation data, it still draws a spot-check sample sized for your target. With evaluation data but no target, the sample is sized for 0.90. With neither, it draws no spot-check.
Ordinal scoring, LLM-as-a-judge, and summarization have no single correct answer to certify against. For ordinal scoring and LLM-as-a-judge, the job reports the measured score. It accepts rows at the fixed 0.85 bar and routes the rest. Summarization has no confidence for each row, so a summarization corpus run routes every summary, and the graders' scores are its quality measure.
On a corpus run, a row that is also a gold row ships with the person's label, not the model's.
Labeling workflow
Most auto-labeling work follows this loop:
-
Split your gold rows – Split the rows that people labeled into development (dev), validation (val), and test sets, and keep the split fixed. Use dev to find mistakes, val to compare specification versions, and test only for the corpus run's certificate.
-
Trial run – Label the dev and val rows without routing, with each set as both the input and the evaluation data. Read the accuracy and the rows that the job got wrong.
-
Refine the specification – Add rules for the mistakes that you see on dev, and run the trial again. Keep the version with the best accuracy on val.
-
Corpus run – Label the full dataset with routing, an accuracy target, and the test rows as evaluation data.
-
Review the routed rows – Have people label the review dataset, including the spot-check rows. For instructions, see Human-in-the-loop annotation.
Start without labeled rows
If you don't have rows that people labeled, start with a cold start. People label a random sample of your data first, and that sample becomes your gold rows:
-
Draw a random sample of about 200 rows from the input data. For a target of 0.95 or higher, draw 400 rows instead. Keep only the
idand the text fields, so that the people who label the sample see no model label. Use a random split, not a stratified one. The certificate assumes that the test rows are a random sample of your data. -
Have people label the sample. For instructions on labeling the sample, see Human-in-the-loop annotation.
-
Split the labeled sample into dev, val, and test sets at random, and then follow the labeling workflow.
If your whole dataset is about the size of the sample, have people label all of it instead; you don't need a labeling job. A corpus run with no evaluation data at all still runs: it accepts rows at the fixed 0.85 bar, certifies nothing, and draws a spot-check only if you set a target.
Write a labeling specification
The job sends your specification to the labeling models before each row, so write it for a model that has never seen your data. Organize it into the following sections:
| Section | What it describes |
|---|---|
Use Case Context |
The task, who uses the labels, and how. |
Quality Standards |
What a good label looks like, such as an accuracy goal and how to handle a row that fits more than one label. |
Input Data |
The fields of each row and what they contain. |
Labelling Configuration |
For classification and named entity recognition, the Label
Set: every label or entity type, each with a one-line
definition. For ordinal scoring and LLM-as-a-judge, Score
Levels instead: one entry for each score, with what earns it.
Then Labelling Rules: the deciding signal for each label,
and how to choose between labels that are easy to confuse. |
Output Schema |
The reply that the job asks for, which you restate. For example,
{"reasoning": "...", "label": "..."} for
classification. |
Keep the following in mind:
-
Make labels mutually exclusive. If two labels can both apply to one row, the models disagree on it and the row is routed. Merge or redefine the labels, or add a rule that says which one wins.
-
Spell labels exactly. The job accepts a label only if it matches a name in the label set.
-
State the scale for a numeric task. For ordinal scoring and LLM-as-a-judge, add a line such as
Scale: 1 to 5. A corpus run of these tasks fails if the specification states no scale. -
Put examples only from your dev rows in the specification. The models read the specification with every row, so a val or test row in it inflates the accuracy that the job measures.
-
Don't refer to the gold label field. The gold labels are for scoring only. The models never see them.
Example: classification
The following specification, shortened, labels customer support messages by intent:
# Customer Support Intent Classification ## Use Case Context Classify incoming customer support messages by intent so that they can be routed to the right team. The labels are used to fine-tune a classification model for real-time ticket routing. ## Quality Standards - Label accuracy of 90% or higher on held-out data - Messages with more than one intent are labeled by their primary intent ## Input Data - message: the text of the support message ## Labelling Configuration ### Label Set | Label | Definition | |---|---| | billing | Charges, invoices, payment methods, refunds, or pricing plans | | technical_issue | Bugs, errors, outages, or unexpected behavior | | account_management | Account settings, users, permissions, or subscriptions | | feature_request | Suggestions for new features or improvements | | general_inquiry | Questions that don't fit the other labels | ### Labelling Rules - billing vs account_management: changing a plan tier that changes the price is billing; changing who has access, with no cost mentioned, is account_management. - feature_request vs technical_issue: if the feature doesn't exist yet, it's feature_request; if it used to work or should work but is broken, it's technical_issue. ## Output Schema {"reasoning": "<one short sentence explaining the label choice>", "label": "<one of the labels in the Label Set>"}
For this specification, an input row and its labeled row look like the
following. The labeled row is shortened: the job also writes fields such as
source and provenance, and on a corpus run the
models' votes.
{"id": "t-1042", "message": "I was charged twice for my March invoice."} {"id": "t-1042", "message": "I was charged twice for my March invoice.", "label": "billing", "reasoning": "The customer reports a duplicate charge.", "confidence": 1.0, "confidence_score": 1.0, "accepted": true, "routed": false}
Inputs
The rows that you label usually come from the step before this one, Data transformation, which you use to deduplicate and filter the data, and to split your gold rows into dev, val, and test sets. You can also bring your own JSON Lines file from Amazon S3.
An auto-labeling job takes the same generation_config as an
SDG job, plus the task options in dataset_options. The following
snippets show only those parts of the job configuration. In the snippets on this page,
replace amzn-s3-demo-bucket with the name of your Amazon S3
bucket. For a complete example, see Use auto-labeling.
- Input data, specification, and models
-
Set
InputDataS3Urito the JSON Lines file of rows to label,SpecS3Urito your labeling specification, andSampleGenerationModelConfigsto the labeling models. Leave outNumSamples: the job labels every row in the input. If you set it, it must equal the number of input rows.generation_config={ "InputDataS3Uri": "s3://amzn-s3-demo-bucket/labeling/tickets.jsonl", "DatasetSpecification": { "SpecS3Uri": "s3://amzn-s3-demo-bucket/labeling/label-spec.md", }, "ModelConfig": { "SampleGenerationModelConfigs": [ {"ModelId": "us.anthropic.claude-sonnet-4-5-20250929-v1:0"}, {"ModelId": "openai.gpt-oss-120b-1:0"}, ] }, } - Task options
-
Pass a task class, such as
ClassificationLabelingOptions, asdataset_options. Every task takes the following:-
text_fields– The input fields that the models read. Each name must be a field of the input rows, and can't be a field that the job writes, such aslabel,confidence, orsplit. -
demos(optional) – ADemosobject whoses3_uripoints to a JSON Lines file of up to 100 labeled example rows that the job adds to every prompt, for exampledemos=Demos(s3_uri="s3://. Draw them from your dev rows, never from the evaluation data. A demo that is also an evaluation row fails the job.amzn-s3-demo-bucket/labeling/demos.jsonl") -
routing(optional) – Runs the job as a corpus run.review_dataset_namenames the review dataset, and must differ from the labeled dataset's name.
Every task except summarization also takes the following:
-
evaluation_data(optional) – The gold rows, as an Amazon S3 JSON Lines file (s3_uri) or a dataset ARN (dataset_arn).label_fieldnames the field that holds each gold label. -
scoring(optional) –targetis the accuracy to certify on the accepted rows, from 0.85 to 0.99.
from sagemaker.ai_registry import ( ClassificationLabelingOptions, EvaluationData, Routing, Scoring) dataset_options=ClassificationLabelingOptions( text_fields=["message"], evaluation_data=EvaluationData( s3_uri="s3://amzn-s3-demo-bucket/labeling/splits/test.jsonl", label_field="gt_label", ), scoring=Scoring(target=0.90), routing=Routing(review_dataset_name="support-tickets-review"), )Summarization takes
gradersinstead of evaluation data and a target: 1–3 models that score a sample of up to 100 summaries, each against its own source text. A grader can't be the labeling model.from sagemaker.ai_registry import Grader, SummarizationLabelingOptions dataset_options=SummarizationLabelingOptions( text_fields=["article"], graders=[Grader(model_id="us.anthropic.claude-sonnet-4-5-20250929-v1:0")], ) -
Prepare the gold rows first. People, not a model, must label the gold rows. For a certificate, the gold rows must be a random sample of the rows that you label. If you have no gold rows yet, see Start without labeled rows.
Use auto-labeling
You can run auto-labeling in the Amazon SageMaker Python SDK directly, or through a coding assistant directed by the core AI and ML skill in the Agent Toolkit for AWS. Both run the same job and produce the same datasets.
Plan for run time and cost. Each row is at least one model call, and a corpus run of a voting task makes one call for each model. Ordinal scoring and LLM-as-a-judge sample one model 5 times for each row. Run trials on your dev and val rows, which are small, before you label the full dataset.
Results
When a job finishes, generate() returns the labeled
DataSet. Its source is the job's
output/ prefix in Amazon S3. The job writes the following files
under its output root, one level above source:
| File | Contents |
|---|---|
output/labels.jsonl |
Every input row with its label, reasoning, and confidence, and whether it was accepted or routed. A row that is a gold row carries the person's label. |
review/review_rows.jsonl |
Corpus runs only. The rows to review: the routed rows, and the
spot-check rows drawn from the accepted rows. Each row holds its
id, its text fields, a split of
low_confidence or spot_check, and a
_review_reason that says why it's in the review set, but not the
model's label, so that reviewers label it without bias. This file is the
review dataset. |
labeling/labeling_summary.json |
The job's summary: its status, row counts, measured accuracy, cutoff,
and certificate. LabelingSummary reads this file. |
labeling/held_out_report.json |
Details of the accuracy on the gold rows, such as the score for each class and the labels that the job confused with each other. |
labeling/failures.jsonl |
Every gold row that the job got wrong, with the gold label and the prediction, plus 2 that it got right for each gold label. For summarization, the summaries that the graders scored low. Use these to refine the specification. |
labeling/review_rows.model_output.jsonl |
Corpus runs only. The model's answers for the review rows, kept apart so that reviewers don't see them. |
labeling/spec.md |
The specification that produced the labels. |
labeling/report.html |
A summary of the job that you can open in a browser. |
labeling/index.json |
The Amazon S3 location of each file in this table. |
The summary's Status and StatusReason tell you what the job
promises about its labels:
| Status | StatusReason | What it means |
|---|---|---|
Certified |
spot_check_pending |
The accepted rows meet your target. The promise holds once people confirm the spot-check rows. If 7 or more of them are wrong, withdraw the certificate. |
NotCertified |
target_not_met |
No cutoff met the target. The job accepted nothing and routed every row except the gold rows. The gold rows keep the person's label. Refine the labeling specification, or lower the accuracy target. |
NotCertified |
insufficient_eval_rows |
The evaluation data had fewer than 50 gold rows to score, so the job accepted nothing and routed every row except the gold rows, which keep the person's label. Add more gold rows. |
NotCertified |
no_eval_data, no_target |
The job had no evaluation data or no target, so it accepted rows at the fixed 0.85 bar. |
NotCertified |
task_not_certifiable |
The task is ordinal scoring, LLM-as-a-judge, or summarization, which the job scores but doesn't certify. |
NotCertified |
refine_round |
The job ran without routing, as a trial. |
Partial |
Any of the reasons in this table | The job couldn't label some rows. Those rows are counted in
RowsMissing, and on a corpus run they're
routed. |
Limits and quotas
The following limits apply to auto-labeling. An auto-labeling job is an SDG job, so it also counts toward the SDG quotas, such as the number of concurrent jobs. For more information about those quotas, see Limits and quotas.
| Limit | Value |
|---|---|
| Rows per job | 50,000, counting the input rows and any gold rows that aren't input rows |
| Input data format | JSON Lines |
| Evaluation data format | JSON Lines, or a JSON array |
| Labeling specification size | 64 KB |
| Demo rows | 100 rows, 64 KB |
Accuracy target (Scoring.Target) |
0.85–0.99 |
| Scored gold rows needed for a certificate | 50 |
| Labeling models on a corpus run | 2 or more for classification, named entity recognition, and question answering. 1 for ordinal scoring and LLM-as-a-judge. |
| Graders for summarization | 1–3 |
For more information about supported Regions and the quotas that apply to every capability, see Regions and limits for SageMaker AI Data Preparation.
Auto-labeling jobs invoke Amazon Bedrock models for every row, so the on-demand quotas for those models, such as tokens per minute and requests per minute, can limit how fast a job runs. For more information, see Quotas for Amazon Bedrock in the Amazon Bedrock User Guide.
Troubleshooting
- The job fails before it labels anything
-
The job checks its inputs before it calls any model, and the error names the field to fix. Common causes:
-
A field in
text_fieldsis missing from the input rows, is repeated, or has the name of a field that the job writes. -
The input data is a CSV, Parquet, or other file that isn't JSON Lines, or the evaluation data is neither JSON Lines nor a JSON array.
-
A corpus run of a voting task lists fewer than 2 models, or a corpus run of ordinal scoring or LLM-as-a-judge lists more than 1.
-
An ordinal scoring or LLM-as-a-judge specification states no scale. Add a line such as
Scale: 1 to 5. -
A classification or named entity recognition specification has no
Label Set. -
NumSamplesis set and doesn't equal the number of input rows. Leave it out. -
On a corpus run with a target, some gold rows can't be scored. This leaves fewer than 50 that can. A gold row is scored when its
idmatches an input row'sid, or when it carries every field intext_fields. Fix the gold rows'idvalues, or add their text fields. If every gold row can be scored but there are fewer than 50, the job runs and ends withinsufficient_eval_rows. -
Two input rows share an
id. A row with noidtakes its position as itsid. -
A demo row is also an evaluation row, by
idor by text.
-
- The job certifies nothing and routes every row
-
The status reason is
target_not_met. Readfailures.jsonlfor the labels that the job confused, add rules for them to the specification, and run a trial on val again. A label set whose labels overlap is the most common cause. - The trial accuracy is high, but the corpus run doesn't certify
-
Check that the test rows are a random sample of the input rows. Gold rows chosen because they were easy, or drawn from another source, measure a higher accuracy than the job gets on the full dataset.
- Most rows are routed even though accuracy is high
-
On a voting task, a row is accepted only when at least 75% of the models agree. Two models that often disagree send many rows to review. Make the label definitions sharper, or choose models that agree more often on your trial.
- The job fails with
HUB_FINALIZATION_FAILED -
SageMaker AI labeled the rows but couldn't register the dataset. The files are still in Amazon S3. Make sure that the execution role has the
sagemaker:ImportHubContentpermission on the hub and on itsDataSethub content. For the required permissions, see Required permissions. - Your session ends before the job finishes
-
The job keeps running.
generate()logs the job's ARN when it submits the job, and a job handle has it asjob.job_arn. To reconnect, replacejob-arnwith the ARN of your job, and runjob = DataPreparer.attach(". Then runjob-arn", wait=False)job.wait(), and readjob.labeling_summary.
Where you go next
The job's output is a labeled dataset, and on a corpus run a review dataset, that you pass by ARN to the next step:
-
Human-in-the-loop annotation – Have people label the review dataset, including the spot-check rows. Then apply their labels to the labeled dataset by
id. -
Data transformation – Filter the accepted rows, or convert the labeled dataset to a training format.
-
Model customization – Fine-tune a model with the labeled dataset as training data.