Preparing Data with SageMaker AI Data Preparation
Customizing or evaluating a model starts with data that fits the task. That data is often missing, too small, unlabeled, or of unknown quality. SageMaker AI Data Preparation is a set of data preparation capabilities that close those gaps. Synthetic data generation, data quality evaluation, data transformation, and auto-labeling run as jobs that SageMaker AI manages. Jobs that produce data register it as a dataset in your account, ready for the next step or for model training. Human-in-the-loop annotation runs in a labeling space where people review and correct labels.
The data preparation journey
The capabilities are stages on one path from no data, or raw data, to a dataset that is ready for training. Stages connect through datasets: you pass a dataset to the next stage by its ARN. You can start at any stage and stop when the data is ready. For example, you can generate records, check their quality, and then split them into training and test sets.
-
Synthetic data generation – Generate records from a dataset specification when you have no data or not enough.
-
Data quality evaluation – Score a dataset against quality criteria before you train on it.
-
Data transformation – Remove duplicates, filter or redact records, split or merge datasets, and convert formats.
-
Auto-labeling – Label records with models, and flag the labels that need human review.
-
Human-in-the-loop annotation – Have people review and correct labels, in chat or in a Label Studio labeling space.
Ways to use data preparation capabilities
You can run synthetic data generation, data quality evaluation, data transformation, and auto-labeling in two ways. Both submit the same jobs and produce the same datasets, so you can switch between them, or use both in one project. Human-in-the-loop annotation runs through a coding assistant.
Amazon SageMaker Python SDK
Use the SDK when you want to script jobs, run them from a notebook or a
pipeline, or control each configuration field. The DataPreparer class
runs the jobs: generate() for synthetic data generation and auto-labeling,
evaluate() for data quality evaluation, and transform() for data transformation. You
need version 3.30.0 or later. To install it and configure your environment, see Get started with SageMaker AI Data Preparation.
Coding assistant with the AI and ML skill
Use a coding assistant when you want to describe what you need in plain language
and be guided through it. The assistant is directed by the core AI and ML skill
(aws-ai-ml) in the Agent Toolkit for AWS. It drafts specifications with
you. It checks your environment and offers to upgrade the SDK when a step needs a
newer version. It asks for your approval before it runs a job or creates resources.
It also saves job runs as a notebook or script that you can rerun. To install the
skill, see the Agent Toolkit for AWS User Guide. You
still need the execution role and default Amazon S3 bucket described in Set up and prerequisites.
Key concepts
Every capability shares the following concepts.
Datasets and hub content
A job's output is a dataset. SageMaker AI writes the
records to Amazon S3 and registers them as hub content of type DataSet in a
hub in your account. In the SDK, a completed job returns a DataSet
object. Its arn identifies the dataset when you pass it to another job,
and its source is the Amazon S3 location of the records. A dataset ARN has
the following format:
arn:aws:sagemaker:region:account-id:hub-content/hub-name/DataSet/dataset-name/version
Jobs that start from an existing dataset take its ARN as input. Data transformation
publishes its output in the same hub as the input, usually as a new version of the
input dataset. Data quality evaluation scores the dataset that you give it and writes the
results to Amazon S3, rather than registering a new dataset. To use
data that you already have, upload it to Amazon S3 and register it as a dataset with
DataSet.create(). If you name a format other than
Custom in dataset_file_format, also pass
validation_role_arn, the IAM role that SageMaker AI uses to validate the
records against that format. Custom skips format validation and needs
no role. For the properties that SageMaker AI records for a dataset, see DataSet properties. For more information about registering data, see Inputs in Data transformation.
Job configuration and schema discovery
You configure a job with a DatasetJobConfig and a
JobCategory, such as SYNTHETIC_DATA_GENERATION. The
fields come from a schema that the SDK gets from the service. Values that you don't
set come from your SDK configuration file, such as the execution role and default
Amazon S3 bucket, and the service applies its own defaults to the fields that remain
unset.
Before it submits a job, the SDK checks that required fields are present and that
values have the right types and structure. The service checks value limits, such as
patterns, lengths, and numeric ranges. When a value is outside a limit, the job
fails, and its FailureReason explains why. To list the fields for a job
type, see Discover configurable fields.