View a markdown version of this page

Preparing Data with SageMaker AI Data Preparation - Amazon SageMaker AI

Preparing Data with SageMaker AI Data Preparation

Customizing or evaluating a model starts with data that fits the task. That data is often missing, too small, unlabeled, or of unknown quality. SageMaker AI Data Preparation is a set of data preparation capabilities that close those gaps. Synthetic data generation, data quality evaluation, data transformation, and auto-labeling run as jobs that SageMaker AI manages. Jobs that produce data register it as a dataset in your account, ready for the next step or for model training. Human-in-the-loop annotation runs in a labeling space where people review and correct labels.

The data preparation journey

The capabilities are stages on one path from no data, or raw data, to a dataset that is ready for training. Stages connect through datasets: you pass a dataset to the next stage by its ARN. You can start at any stage and stop when the data is ready. For example, you can generate records, check their quality, and then split them into training and test sets.

Ways to use data preparation capabilities

You can run synthetic data generation, data quality evaluation, data transformation, and auto-labeling in two ways. Both submit the same jobs and produce the same datasets, so you can switch between them, or use both in one project. Human-in-the-loop annotation runs through a coding assistant.

Amazon SageMaker Python SDK

Use the SDK when you want to script jobs, run them from a notebook or a pipeline, or control each configuration field. The DataPreparer class runs the jobs: generate() for synthetic data generation and auto-labeling, evaluate() for data quality evaluation, and transform() for data transformation. You need version 3.30.0 or later. To install it and configure your environment, see Get started with SageMaker AI Data Preparation.

Coding assistant with the AI and ML skill

Use a coding assistant when you want to describe what you need in plain language and be guided through it. The assistant is directed by the core AI and ML skill (aws-ai-ml) in the Agent Toolkit for AWS. It drafts specifications with you. It checks your environment and offers to upgrade the SDK when a step needs a newer version. It asks for your approval before it runs a job or creates resources. It also saves job runs as a notebook or script that you can rerun. To install the skill, see the Agent Toolkit for AWS User Guide. You still need the execution role and default Amazon S3 bucket described in Set up and prerequisites.

Key concepts

Every capability shares the following concepts.

Datasets and hub content

A job's output is a dataset. SageMaker AI writes the records to Amazon S3 and registers them as hub content of type DataSet in a hub in your account. In the SDK, a completed job returns a DataSet object. Its arn identifies the dataset when you pass it to another job, and its source is the Amazon S3 location of the records. A dataset ARN has the following format:

arn:aws:sagemaker:region:account-id:hub-content/hub-name/DataSet/dataset-name/version

Jobs that start from an existing dataset take its ARN as input. Data transformation publishes its output in the same hub as the input, usually as a new version of the input dataset. Data quality evaluation scores the dataset that you give it and writes the results to Amazon S3, rather than registering a new dataset. To use data that you already have, upload it to Amazon S3 and register it as a dataset with DataSet.create(). If you name a format other than Custom in dataset_file_format, also pass validation_role_arn, the IAM role that SageMaker AI uses to validate the records against that format. Custom skips format validation and needs no role. For the properties that SageMaker AI records for a dataset, see DataSet properties. For more information about registering data, see Inputs in Data transformation.

Job configuration and schema discovery

You configure a job with a DatasetJobConfig and a JobCategory, such as SYNTHETIC_DATA_GENERATION. The fields come from a schema that the SDK gets from the service. Values that you don't set come from your SDK configuration file, such as the execution role and default Amazon S3 bucket, and the service applies its own defaults to the fields that remain unset.

Before it submits a job, the SDK checks that required fields are present and that values have the right types and structure. The service checks value limits, such as patterns, lengths, and numeric ranges. When a value is outside a limit, the job fails, and its FailureReason explains why. To list the fields for a job type, see Discover configurable fields.