MLPER-06: Explore alternatives for performance improvement

Perform benchmarks to improve the machine learning model performance. Benchmarking in ML involves evaluation and comparison of ML workloads with different algorithms, features, and architecture resources. It enables identifying the combination with optimal performance.

Options you can use when benchmarking include:

• Use more data to broaden the statistical range and improve the success metric of the model.

• Apply feature engineering to extract important signals in the data for the model.

• Make alternative algorithm selections for an optimal fit to the specifics of the data.

• Ensemble methods that combine the different advantages of multiple models.

• Tune the hyperparameters for a given algorithm to calibrate the model for the data.

Implementation plan

• Use Amazon SageMaker Experiments to optimize algorithms and features -Begin with a simple architecture, obvious features, and a simple algorithm to establish a baseline. Amazon SageMaker provides built-in algorithms for developing a baseline model. Use Amazon SageMaker Experiments to organize, track, compare, and evaluate your machine learning experiments. Test different algorithms with increasing complexity to observe performance. Combine models into an ensemble to increase accuracy, but consider the potential loss of efficiency as a trade-off. Refine the features by selection and modify parameters to optimize model performance. Tune the model’s hyperparameters to optimize performance using Amazon SageMaker Hyperparameter Optimization to automate the search.

Documents

• Improving Model Accuracy

• Evaluating ML Models

• Feature Processing with Spark ML and Scikit-learn

• Perform Automatic Model Tuning with SageMaker

• Amazon SageMaker Experiments: Track and Compare Tutorial

Blogs

• Running multiple HPO jobs in parallel on Amazon SageMaker

• Optimizing portfolio value with Amazon SageMaker automatic model tuning

• Utilizing XGBoost training reports to improve your models

Videos

• Tune your ML models to the highest accuracy with automatic model tuning

• Organize, Track, and Evaluate ML Training Runs With Amazon SageMaker Experiments

Examples

• Feature Engineering Immersion Day Workshop

• Improving Forecast Accuracy with Machine Learning

• Ensemble Predictions From Multiple Models