Amazon SageMaker Model Cards - Amazon SageMaker

Amazon SageMaker Model Cards

Use Amazon SageMaker Model Cards to document critical details about your machine learning (ML) models in a single place for streamlined governance and reporting.

Catalog details such as the intended use and risk rating of a model, training details and metrics, evaluation results and observations, and additional call-outs such as considerations, recommendations, and custom information. By creating model cards, you can do the following:

  • Provide guidance on how a model should be used.

  • Support audit activities with detailed descriptions of model training and performance.

  • Communicate how a model is intended to support business goals.

Model cards provide prescriptive guidance on what information to document and include fields for custom information. After creating a model card, you can export it to a PDF or download it to share with relevant stakeholders. Any edits other than an approval status update made to a model card result in additional model card versions in order to have an immutable record of model changes.

Prerequisites

To get started with Amazon SageMaker Model Cards, you must have permission to create, edit, view, and export model cards.

Intended uses of a model

Specifying the intended uses of a model helps ensure that model developers and users have the information they need to train or deploy the model responsibly. The intended uses of a model should describe the scenarios in which the model is appropriate to use as well as the scenarios in which the model is not recommended to use.

We recommend including:

  • The general purpose of the model

  • Use cases for which the model was intended

  • Use cases for which the model was not intended

  • Assumptions made when developing the model

The intended uses of a model go beyond technical details and describe how a model should be used in production, the scenarios in which is appropriate to use a model, and additional considerations such as the type of data to use with the model or any assumptions made during development.

Risk ratings

Developers create ML models for use cases with varying levels of risk. For example, a model that approves loan applications might be a higher risk model than one that detects the category of an email. Given the varied risk profiles of a model, model cards provide a field for you to categorize a model’s risk rating.

This risk rating can be unknown, low, medium, or high. Use these risk rating fields to label unknown, low, medium, or high-risk models and help your organization comply with any existing rules about putting certain models into production.

Model card JSON schema

Evaluation details for a model card must be provided in JSON format. If you have existing JSON format evaluation reports generated by SageMaker Clarify or SageMaker Model Monitor, upload them to Amazon S3 and provide an S3 URI to automatically parse evaluation metrics. For more information and sample reports, see the example metrics folder in the Amazon SageMaker Model Governance - Model Cards example notebook.

When creating a model card using the SageMaker Python SDK, model content must be in the model card JSON schema and provided as a string. Provide model content similar to the following example.

{ "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://json-schema.org/draft-07/schema#", "title": "SageMakerModelCardSchema", "description": "Default model card schema", "version": "0.1.0", "type": "object", "additionalProperties": false, "properties": { "model_overview": { "description": "Overview about the model", "type": "object", "additionalProperties": false, "properties": { "model_description": { "description": "description of model", "type": "string", "maxLength": 1024 }, "model_owner": { "description": "Owner of model", "type": "string", "maxLength": 1024 }, "model_creator": { "description": "Creator of model", "type": "string", "maxLength": 1024 }, "problem_type": { "description": "Problem being solved with the model", "type": "string" }, "algorithm_type": { "description": "Algorithm used to solve the problem", "type": "string", "maxLength": 1024 }, "model_id": { "description": "SageMaker Model ARN or non-SageMaker Model ID", "type": "string", "maxLength": 1024 }, "model_artifact": { "description": "Location of model artifacts", "type": "array", "maxContains": 15, "items": { "type": "string", "maxLength": 1024 } }, "model_name": { "description": "Name of the model", "type": "string", "maxLength": 1024 }, "model_version": { "description": "Version of the model", "type": "number", "minimum": 1 }, "inference_environment": { "description": "Overview about model inference", "type": "object", "additionalProperties": false, "properties": { "container_image": { "description": "SageMaker inference image URI", "type": "array", "maxContains": 15, "items": { "type": "string", "maxLength": 1024 } } } } } }, "intended_uses": { "description": "Intended usage of model", "type": "object", "additionalProperties": false, "properties": { "purpose_of_model": { "description": "General purpose of model", "type": "string", "maxLength": 2048 }, "intended_uses": { "description": "Intended use cases", "type": "string", "maxLength": 2048 }, "factors_affecting_model_efficiency": { "type": "string", "maxLength": 2048 }, "risk_rating": { "description": "Risk rating for model card", "$ref": "#/definitions/risk_rating" }, "explanations_for_risk_rating": { "type": "string", "maxLength": 2048 } } }, "training_details": { "description": "Overview about model training", "type": "object", "additionalProperties": false, "properties": { "objective_function": { "description": "The objective function that the model optimizes", "function": { "$ref": "#definitions/objective_function" }, "notes": { "type": "string", "maxLength": 1024 } }, "training_observations": { "type": "string", "maxLength": 1024 }, "training_job_details": { "type": "object", "additionalProperties": false, "properties": { "training_arn": { "description": "SageMaker training job ARN", "type": "string", "maxLength": 1024 }, "training_datasets": { "description": "Location of the model datasets", "type": "array", "maxContains": 15, "items": { "type": "string", "maxLength": 1024 } }, "training_environment": { "type": "object", "additionalProperties": false, "properties": { "container_image": { "description": "SageMaker training image URI", "type": "array", "maxContains": 15, "items": { "type": "string", "maxLength": 1024 } } } }, "training_metrics": { "type": "array", "items": { "maxItems": 50, "$ref": "#/definitions/training_metric" } }, "user_provided_training_metrics": { "type": "array", "items": { "maxItems": 50, "$ref": "#/definitions/training_metric" } } } } } }, "evaluation_details": { "type": "array", "default": [], "items": { "type": "object", "required": [ "name" ], "additionalProperties": false, "properties": { "name": { "type": "string", "pattern": ".{1,63}" }, "evaluation_observation": { "type": "string", "maxLength": 2096 }, "evaluation_job_arn": { "type": "string", "maxLength": 256 }, "datasets": { "type": "array", "items": { "type": "string", "maxLength": 1024 }, "maxItems": 10 }, "metadata": { "description": "Additional attributes associated with the evaluation results", "type": "object", "additionalProperties": { "type": "string", "maxLength": 1024 } }, "metric_groups": { "type": "array", "default": [], "items": { "type": "object", "required": [ "name", "metric_data" ], "properties": { "name": { "type": "string", "pattern": ".{1,63}" }, "metric_data": { "type": "array", "items": { "anyOf": [ { "$ref": "#/definitions/simple_metric" }, { "$ref": "#/definitions/linear_graph_metric" }, { "$ref": "#/definitions/bar_chart_metric" }, { "$ref": "#/definitions/matrix_metric" } ] } } } } } } } }, "additional_information": { "additionalProperties": false, "type": "object", "properties": { "ethical_considerations": { "description": "Any ethical considerations that the model card author wants to document", "type": "string", "maxLength": 2048 }, "caveats_and_recommendations": { "description": "Caveats and recommendations for those who might use this model in their applications.", "type": "string", "maxLength": 2048 }, "custom_details": { "type": "object", "additionalProperties": { "$ref": "#/definitions/custom_property" } } } } }, "definitions": { "risk_rating": { "description": "Your organization's risk rating of the model", "type": "string", "enum": [ "High", "Medium", "Low", "Unknown" ] }, "custom_property": { "description": "Additional property in section", "type": "string", "maxLength": 1024 }, "objective_function": { "description": "The objective function that the training job optimizes", "additionalProperties": false, "properties": { "function": { "type": "string", "enum": [ "Maximize", "Minimize" ] }, "facet": { "type": "string", "maxLength": 63 }, "condition": { "type": "string", "maxLength": 63 } } }, "training_metric": { "description": "Training metric data", "type": "object", "required": [ "name", "value" ], "additionalProperties": false, "properties": { "name": { "type": "string", "pattern": ".{1,255}" }, "notes": { "type": "string", "maxLength": 1024 }, "value": { "type": "number" } } }, "linear_graph_metric": { "type": "object", "required": [ "name", "type", "value" ], "additionalProperties": false, "properties": { "name": { "type": "string", "pattern": ".{1,255}" }, "notes": { "type": "string", "maxLength": 1024 }, "type": { "type": "string", "enum": [ "linear_graph" ] }, "value": { "anyOf": [ { "type": "array", "items": { "type": "array", "items": { "type": "number" }, "minItems": 2, "maxItems": 2 }, "minItems": 1, "maxItems": 20 } ] }, "x_axis_name": { "$ref": "#/definitions/axis_name_string" }, "y_axis_name": { "$ref": "#/definitions/axis_name_string" } } }, "bar_chart_metric": { "type": "object", "required": [ "name", "type", "value" ], "additionalProperties": false, "properties": { "name": { "type": "string", "pattern": ".{1,255}" }, "notes": { "type": "string", "maxLength": 1024 }, "type": { "type": "string", "enum": [ "bar_chart" ] }, "value": { "anyOf": [ { "type": "array", "items": { "type": "number" }, "minItems": 1, "maxItems": 20 } ] }, "x_axis_name": { "$ref": "#/definitions/axis_name_array" }, "y_axis_name": { "$ref": "#/definitions/axis_name_string" } } }, "matrix_metric": { "type": "object", "required": [ "name", "type", "value" ], "additionalProperties": false, "properties": { "name": { "type": "string", "pattern": ".{1,255}" }, "notes": { "type": "string", "maxLength": 1024 }, "type": { "type": "string", "enum": [ "matrix" ] }, "value": { "anyOf": [ { "type": "array", "items": { "type": "array", "items": { "type": "number" }, "minItems": 1, "maxItems": 20 }, "minItems": 1, "maxItems": 20 } ] }, "x_axis_name": { "$ref": "#/definitions/axis_name_array" }, "y_axis_name": { "$ref": "#/definitions/axis_name_array" } } }, "simple_metric": { "description": "metric data", "type": "object", "required": [ "name", "type", "value" ], "additionalProperties": false, "properties": { "name": { "type": "string", "pattern": ".{1,255}" }, "notes": { "type": "string", "maxLength": 1024 }, "type": { "type": "string", "enum": [ "number", "string", "boolean" ] }, "value": { "anyOf": [ { "type": "number" }, { "type": "string", "maxLength": 63 }, { "type": "boolean" } ] }, "x_axis_name": { "$ref": "#/definitions/axis_name_string" }, "y_axis_name": { "$ref": "#/definitions/axis_name_string" } } }, "axis_name_array": { "type": "array", "items": { "type": "string", "maxLength": 63 } }, "axis_name_string": { "type": "string", "maxLength": 63 } } }