MLPER-07: Establish a model performance evaluation pipeline

Capture key metrics related to model performance using an end-to-end performance pipeline to evaluate the success of a model. Choose specific metrics based on the use case and the business KPIs. Sample key metrics include training or validation errors, and prediction accuracy. Specific model performance metrics include Root Mean Squared Error (RMSE), accuracy, precision, recall, F1 score, and area under the curve (AUC). Establish a fully automated performance testing pipeline system to initiate evaluation every time there is an updated model or data.

Implementation plan

• Create an end-to-end workflow with Amazon SageMaker Pipelines - Start with a workflow template to establish an initial infrastructure for model training and deployment. SageMaker Pipelines helps you automate different steps of the ML workflow. These steps include data loading, data transformation, training, tuning, and deployment. With SageMaker Pipelines, you can share and reuse workflows to re-create or optimize models, helping you scale ML throughout your organization. Within SageMaker Pipelines, the SageMaker Model Registry tracks the model versions and respective artifacts. These artifacts include the metadata and lineage data collected throughout the model development lifecycle. SageMaker Model Registry can also enable automating model deployment with CI/CD.

Documents

• Define a Pipeline

Blogs

• Building, automating, managing, and scaling ML workflows using Amazon SageMaker Pipelines

• Extend Amazon SageMaker Pipelines to include custom steps using callback steps

Videos

• Introducing Amazon SageMaker Pipelines

• How to create fully automated ML workflows with Amazon SageMaker Pipelines

Examples

• SageMaker Pipelines – Immersion Day