Amazon SageMaker Model Monitor - Amazon SageMaker

Amazon SageMaker Model Monitor

Amazon SageMaker Model Monitor continuously monitors the quality of Amazon SageMaker machine learning models in production. It enables developers to set alerts for when there are deviations in the model quality, such as data drift. Early and pro-active detection of these deviations enables you to take corrective actions, such as retraining models, auditing upstream systems, or fixing data quality issues without having to monitor models manually or build additional tooling. You can use Model Monitor pre-built monitoring capabilities that do not require coding. You also have the flexibility to monitor models by coding to provide custom analysis.

How Model Monitor Works

Amazon SageMaker Model Monitor automatically monitors machine learning (ML) models in production and notifies you when data quality issues arise. ML models in production have to make predictions on real-life data that is not carefully curated like most training datasets. If the statistical nature of the data that your model receives while in production drifts away from the nature of the baseline data it was trained on, the model begins to lose accuracy in its predictions. Model Monitor uses rules to detect data drift and alerts you when it happens. The following figure shows how this process works.


                The model monitoring process with Amazon SageMaker Model Monitor.

To enable model monitoring, you take the following steps, which follow the path of the data through the various data collection, monitoring, and analysis processes.

  • Capture Data: Enable the endpoint to capture data from incoming requests to a trained ML model and the resulting model predictions.

  • Create a Baseline: Create a baseline from the dataset that was used to train the model. Compute baseline schema constraints and statistics for each feature using Deequ, an open source library built on Apache Spark, which is used to measure data quality in large datasets.

  • Schedule Monitoring Jobs: Create a monitoring schedule specifying what data to collect, how often to collect it, how to analyze it, and which reports to produce.

  • Interpret Results: Inspect the reports, which compare the latest data with the baseline, and watch for any violations reported and for metrics and notifications from Amazon CloudWatch.

Note

Amazon SageMaker Model Monitor currently supports only endpoints that host a single model and does not support monitoring multi-model endpoints. For information on using multi-model endpoints, see Host Multiple Models with Multi-Model Endpoints .

Model Monitor Sample Notebooks

For a sample notebook that takes you through the full end-to-end workflow for Model Monitor, see the Introduction to Amazon SageMaker Model Monitor.

For a sample notebook that enables the model monitoring experience for an existing endpoint, see the Enable Model Monitoring.

For a sample notebook that visualizes the statistics.json file for a selected execution in a monitoring schedule, see the Model Monitor Visualization.

For instructions how to create and access Jupyter notebook instances that you can use to run the example in Amazon SageMaker, see Use Amazon SageMaker Notebook Instances. Once you have created a notebook instance and opened it, choose the SageMaker Examples tab to see a list of all the Amazon SageMaker samples. To open a notebook, choose its Use tab and choose Create copy.