MLPER-09: Perform a performance trade-off analysis

Perform alternative trade-off analysis to obtain optimal performance and accuracy for a given use-case data and business requirement.

• Accuracy versus complexity trade-off: The simpler a machine learning model is, the more explainable are its predictions. Deep learning predictions can potentially outperform linear regression or a decision tree algorithm, but at the cost of added complexity in interpretability and explainability.

• Bias versus fairness trade-off: Define a process for managing risks of bias and fairness in model performance. Business value most often aligns with models that have considered historical or sampling biases in the training data. Further consideration should be given to the disparate impact of inaccurate model predictions. For example, underrepresented groups are often more impacted by historical biases, which might perpetuate unfair practices.

• Bias versus variance trade-off (supervised ML): The goal is to achieve a trained model with the lowest bias versus variance tradeoff for a given data set. To help overcome bias and variance errors, you can use:

  • Cross validation

  • More data

  • Regularization

  • Simpler models

  • Dimension reduction (Principal Component Analysis)

  • Stop training early

• Precision versus recall trade-off (supervised ML): This analysis can be important when precision is more important than recall or vice versa. For example, optimization of precision is more important when the goal is to reduce false positives. However, optimization of recall is more important when the goal is to reduce false negatives. It’s not possible to have both high precision and high recall-if one is increased, the other decreases. A trade-off analysis helps identify the optimal option for analysis.

Implementation plan

Construct alternate workflows to optimize all aspects of business value - Complex models can deliver high accuracy. If the business requirements include low-latency, then the model might need to be simplified with lower complexity. Identify how trade-offs affect accuracy and the latency of inferences. Test these trade-offs using Amazon SageMaker Experiments to keep track of each model type. Amazon SageMakerClarify provides explanations of the data, models, and monitoring used to assess predictions. It can measure biases during each stage of the ML lifecycle. Provided explanations will help understanding how fairness affects the business use case. Complex ML models can be slower to return an inference and more difficult to deploy at the edge. Amazon SageMaker Neo enables developers to optimize ML models for inference on SageMaker in the cloud and supported devices at the edge.

Documents

• Evaluating ML Models

• AI Fairness and Explainability Whitepaper

• Optimize model performance using Amazon SageMaker Neo

Blogs

• Amazon SageMaker Experiments – Organize, Track And Compare Your Machine Learning Trainings

• Amazon SageMaker Clarify Detects Bias and Increases the Transparency of Machine Learning Models

• Unlock near 3x performance gains with XGBoost and Amazon SageMaker Neo

Videos

• Machine learning and society: Bias, fairness, and explainability