MLPER-03: Define relevant evaluation metrics
To validate and monitor model performance, establish numerical metrics that directly relate to the KPIs. These KPIs are established in the business goal identification phase. Evaluate whether the performance metrics accurately reflect the business’ tolerance for the error. For instance, false positives might lead to excessive maintenance costs in predictive maintenance use cases. Numerical metrics, such as precision and recall, would help differentiate the business requirements and be closer aligned to business value. Consider developing custom metrics that tune the model directly for the business objectives. Examples of standard metrics for ML models include:
-
Classification
-
Confusion matrix (precision, recall, accuracy, F1 score)
-
Receiver operating characteristic-area under curve (AUC)
-
Logarithmic loss (log-loss)
-
-
Regression
-
Root mean square error (RMSE)
-
Mean absolute percentage error (MAPE)
-
Implementation plan
-
Optimize business-related metrics - Identify performance metrics relevant to use-case and model type. Implement the metric as a loss function or use the loss function included in Amazon SageMaker
. Use Amazon SageMaker Experiments to evaluate the metrics with consideration to the business use case to maximize business value. Track model and concept drift in real time with Amazon SageMaker Model Monitor to estimate errors. -
Calculate the maximum probability of error that will be required for the ML model to produce results considering the tolerance set by the business.
-
Select and train ML models on the available data to make prediction within the probability bounds. Organize tests on different models with Amazon SageMaker Experiments.
-
Documents
Blogs
Videos
Examples
