View a markdown version of this page

DataSet properties - Amazon SageMaker AI

DataSet properties

The HubContentDocument for the DataSet HubContentType contains the metadata that describes a dataset stored in a SageMaker hub, including its file format, Amazon S3 location, validation results, and generation and job metadata, for both bring your own data (BYOD) and synthetic data generation (SDG) datasets. Use this metadata to evaluate and compare datasets and to make informed decisions about which dataset best fits your use case.

When calling the DescribeHubContent operation for a dataset, the response includes a HubContentDocument field, returned as a JSON-formatted string. This section describes the properties of that document for the DataSet HubContentType so that it can be more easily interpreted.

Parse the HubContentDocument string as JSON to read the properties described here. Property names use PascalCase. A document contains only the properties that apply to the dataset, so most properties are optional.

The DatasetFileFormat property is always required. For a BYOD dataset (when DatasetType is not SyntheticDataGeneration), DatasetS3Bucket and DatasetS3Prefix are also required. All other properties are optional and appear only when they apply to the dataset.

Creating a dataset triggers automatic data validation

When you create a dataset, SageMaker automatically runs a DataQualityEvaluation job that validates the data against the declared DatasetFileFormat. SageMaker performs this validation at no charge to you.

Top-level properties

The following table lists the top-level properties of the dataset HubContentDocument.

Property

Type

Description

DatasetFileFormat

enum

The format of the dataset files. Required for all dataset types. Valid values: Bedrock_Converse, HF_Prompt_Completion, HF_Preference, Verl, Nova_PPO, Nova_CPT, OpenAI_Messages, SageMaker_Eval, Nova_RLVR, Nova_DPO, MTRL, Custom.

Required

DatasetType

enum

An optional type discriminator. Valid values are SyntheticDataGeneration or BYOD.

Optional

DatasetS3Bucket

string

The S3 bucket name. Required for a BYOD dataset (when DatasetType is not SyntheticDataGeneration).

Required for BYOD

DatasetS3Prefix

string

The S3 prefix path within the bucket. Required for a BYOD dataset (when DatasetType is not SyntheticDataGeneration).

Required for BYOD

DatasetS3Uri

S3Uri

The S3 URI of the generated dataset file.

Optional

SpecS3Uri

S3Uri

The S3 URI of the dataset specification document.

Optional

ValidationRoleArn

IamRoleArn

The IAM role used to run dataset validation.

Optional

ValidationReportS3Uri

S3Uri

The S3 URI of the validation report.

Optional

TransformationReportS3Uri

S3Uri

The S3 URI of the transformation execution report.

Optional

TransformationCodeS3Uri

S3Uri

The S3 URI of the generated transformation code (transformer.py) produced by a DataTransformation code-generation job.

Optional

CodeArtifacts

object

The generator and verifier code artifacts.

Optional

NumRequestedSamples

int >= 1

The number of samples requested for generation.

Optional

NumValidSamples

int >= 0

The number of samples that passed validation.

Optional

DatasetJobArn

string

The ARN of the synthetic data generation (SDG) job that created this dataset.

Optional

TransformationJobArn

string

The ARN of the DataTransformation job that created this dataset.

Optional

MlflowConfig

object

The MLflow configuration.

Optional

SyntheticDataGenerationJobMetadata

object

The generation history and job configuration.

Optional

DataQualityMetadata

object[]

An array of data quality evaluation metadata entries. Each entry points to an index file (MetadataS3Uri) that lists the individual report URIs.

Optional

JobMetadata

object[]

An array of dataset job metadata entries, one for each dataset job that produced or evaluated this dataset. Each entry points to an index file with that job's output report URIs; it records which job wrote it but is not service-attested.

Optional

ValidationManifest

object

The results of automatic dataset validation, including record counts, failure details, and detected features. Read-only: the service populates it when validation runs; you cannot set it directly.

Optional

ContentMetadata

object

Metadata for datasets extracted from SageMaker Feature Store, used for lineage tracking between feature groups and training jobs.

Optional

Metadata

object

Optional key-value pairs that describe the dataset. Keys are 1–128 characters; each value is a string, number, boolean, object, or an array of those. Up to 50 properties.

Optional

Nested properties

The following table lists the properties nested inside the object and array properties in the preceding table. A nested property marked as required is required only when its parent property is present.

Property path

Type

Description

CodeArtifacts.GenerationCodeS3Uri

S3 URI

The S3 URI of the generator script.

Required

CodeArtifacts.VerifiersCodeS3Uri

S3 URI

The S3 URI of the verifier script.

Required

MlflowConfig.MlflowResourceArn

string

The ARN of the MLflow app resource.

Required

MlflowConfig.MlflowExperimentName

string

The name of the MLflow experiment associated with this dataset.

Required

SyntheticDataGenerationJobMetadata.InputDataS3Uri

S3 URI

The S3 URI of the seed/contextual data and documents used for generation.

Optional

SyntheticDataGenerationJobMetadata.SampleGenerationStrategy

enum RoundRobin | Jury

How the sample generation models were combined. RoundRobin (default) assigns each sample to one model; Jury sends every sample to all models and keeps the majority response. Default is RoundRobin.

Optional

SyntheticDataGenerationJobMetadata.ModelConfig

object

The large language model (LLM) configuration for generation and verification. Each model config accepts a model ID hosted on Amazon Bedrock.

Optional

SyntheticDataGenerationJobMetadata.ModelConfig.SampleGenerationModelConfigs

array

Configurations for the LLMs used for sample generation (1–5). When multiple models are specified, generation requests are distributed across them in round-robin.

Optional

SyntheticDataGenerationJobMetadata.ModelConfig.SampleGenerationModelConfigs[].ModelId

string

The Bedrock model ID used for sample generation.

Required

SyntheticDataGenerationJobMetadata.ModelConfig.SampleGenerationModelConfigs[].TargetTokensPerMinute

integer

The best-effort target token-per-minute throughput limit for sample generation; actual usage might briefly exceed it.

Optional

SyntheticDataGenerationJobMetadata.ModelConfig.SampleVerificationModelConfigs

array

Configurations for the LLMs used for verification (1–10).

Optional

SyntheticDataGenerationJobMetadata.ModelConfig.SampleVerificationModelConfigs[].ModelId

string

The Bedrock model ID used for verification.

Optional

SyntheticDataGenerationJobMetadata.ModelConfig.SampleVerificationModelConfigs[].TargetTokensPerMinute

integer

The best-effort target token-per-minute throughput limit for verification.

Optional

SyntheticDataGenerationJobMetadata.ModelConfig.CodeGenerationModelConfig

object

The configuration for the LLM used for code generation. Only Anthropic Claude Sonnet and Opus models are supported.

Optional

SyntheticDataGenerationJobMetadata.ModelConfig.CodeGenerationModelConfig.ModelId

string

The Bedrock model ID used for code generation.

Optional

SyntheticDataGenerationJobMetadata.ModelConfig.CodeGenerationModelConfig.TargetTokensPerMinute

integer

The best-effort target token-per-minute throughput limit for code generation.

Optional

SyntheticDataGenerationJobMetadata.SyntheticDataGenerationEngineVersion

string

The version of the generation engine.

Optional

SyntheticDataGenerationJobMetadata.CreatedAt

string (date-time)

The timestamp when the generation job completed.

Optional

SyntheticDataGenerationJobMetadata.CustomerFacingMessage

string

A human-readable summary of the generation outcome.

Optional

DataQualityMetadata[].MetadataS3Uri

S3 URI

The S3 URI of the data quality index file that lists the individual report URIs (aggregate, record-level, recommendations).

Required

DataQualityMetadata[].CreatedAt

string (date-time)

The timestamp when the data quality evaluation was created.

Required

DataQualityMetadata[].EvaluationName

string

A human-readable name for this evaluation run.

Required

JobMetadata[].MetadataS3Uri

S3 URI

The S3 URI of the job's index file that lists its individual report URIs.

Required

JobMetadata[].CreatedAt

string (date-time)

The timestamp when the job completed.

Required

JobMetadata[].JobName

string

The name of the dataset job that wrote this entry.

Required

JobMetadata[].JobType

enum SyntheticDataGeneration | DataQualityEvaluation | DataTransformation | GapAnalysis

The category of the dataset job that wrote this entry.

Required

ValidationManifest.JobStatus

enum IN_PROGRESS | COMPLETED | ERROR | SKIPPED

The status of the validation run. IN_PROGRESS: Validation is running. COMPLETED: Results are available. ERROR: Validation could not complete (see FailureReason and Retryable). SKIPPED: No validation job ran for this version.

Required

ValidationManifest.Retryable

boolean

Specifies whether the error is transient and a retry might succeed. Applies when JobStatus is ERROR; if false, the error requires customer action.

Optional

ValidationManifest.FailureReason

string

Present when JobStatus is ERROR. A human-readable description of the scan-level condition that prevented validation.

Optional

ValidationManifest.ValidatedAt

string (date-time)

The timestamp when validation completed.

Optional

ValidationManifest.ValidationJobArn

string

The ARN of the validation job that produced this result.

Optional

ValidationManifest.Modality

enum TEXT

The data modality of the dataset.

Optional

ValidationManifest.RecordCount

integer

The total number of records in the dataset.

Optional

ValidationManifest.ValidRecords

integer

The number of records that passed validation.

Optional

ValidationManifest.FailedRecords

integer

The number of records that failed validation.

Optional

ValidationManifest.FailureRate

number

The fraction of records that failed validation (FailedRecords/RecordCount), from 0 to 1.

Optional

ValidationManifest.SummaryByReason

array

Failure counts grouped by reason code.

Optional

ValidationManifest.SummaryByReason[].ReasonCode

string

The code identifying the type of validation failure.

Required

ValidationManifest.SummaryByReason[].Message

string

A human-readable description of the failure.

Optional

ValidationManifest.SummaryByReason[].Count

integer

The number of records with this failure.

Required

ValidationManifest.Features

object

Detected dataset features such as schema shape, field presence rates, and structure observations.

Optional

ValidationManifest.FailedRecordIndices

object

The indices of records that failed validation, with an inline preview and an S3 reference to the full list.

Optional

ValidationManifest.FailedRecordIndices.S3Uri

S3 URI

The S3 URI of the complete list of failed record indices.

Optional

ValidationManifest.FailedRecordIndices.Count

integer

The total number of failed record indices.

Optional

ValidationManifest.FailedRecordIndices.Preview

integer[]

An inline preview of failed record indices (up to 500).

Optional

ValidationManifest.FailedRecordIndices.PreviewTruncated

boolean

Specifies whether the full list exceeds the inline preview limit.

Optional

ValidationManifest.FailedRecordIndices.FailuresS3Uri

S3 URI

The S3 URI of per-record failure details.

Optional

ContentMetadata.SourceFeatureGroups

string[]

The ARNs of the feature groups that contributed data to this dataset extraction (1–50).

Required

ContentMetadata.ExtractionMethod

enum FeatureStoreDatasetBuilder

The method used to extract data from Feature Store.

Optional

ContentMetadata.AthenaQueryExecutionId

string

The Athena query execution ID that produced the dataset CSV.

Optional

Metadata[key]

string | number | boolean | object | array

A single key-value entry describing the dataset. Keys are 1–128 characters.

Optional