Best practices by ML lifecycle phase

There are six phases in the machine learning lifecycle and within each there are six pillars of the Well-Architected Framework. The following index lists the best practices by lifecycle phase.

Business goal identification phase

Operational excellence pillar

• MLOE-01: Develop the right skills with accountability and empowerment

• MLOE-02: Discuss and agree on the level of model explainability

• MLOE-03: Monitor model compliance to business requirements

Security pillar

• MLSEC-01: Validate ML data permissions, privacy, software, and license terms

Reliability pillar

There are no reliability pillar best practices for business goal identification.

Performance efficiency pillar

• MLPER-01: Determine key performance indicators

Cost optimization pillar

• MLCOST-01: Define overall return on investment (ROI) and opportunity cost

• MLCOST-02: Use managed services to reduce total cost of ownership (TCO)

Sustainability pillar

• MLSUS-01: Define the overall environmental impact or benefit

ML problem framing phase

Operational excellence pillar

• MLOE-04: Establish ML roles and responsibilities

• MLOE-05: Prepare an ML profile template

• MLOE-06: Establish model improvement strategies

• MLOE-07: Establish a lineage tracker system

• MLOE-08: Establish feedback loops across ML lifecycle phases

• MLOE-09: Review fairness and explainability

Security pillar

• MLSEC-02: Design data encryption and obfuscation 

Reliability pillar

• MLREL-01: Use APIs to abstract change from model consuming applications

• MLREL-02: Adopt a machine learning microservice strategy

Performance efficiency pillar

• MLPER-02: Use purpose-built AI and ML services and resources

• MLPER-03: Define relevant evaluation metrics

Cost optimization pillar

• MLCOST-03: Identify if machine learning is the right solution

• MLCOST-04: Tradeoff analysis on custom versus pre-trained models

Sustainability pillar

• MLSUS-02: Consider AI services and pre-trained models

• MLSUS-03: Select sustainable Regions

Data processing phase

Operational excellence pillar

• MLOE-10: Profile data to improve quality

• MLOE-11: Create tracking and version control mechanisms

Security pillar

• MLSEC-03: Ensure least privilege access

• MLSEC-04: Secure data and modeling environment

• MLSEC-05: Protect sensitive data privacy

• MLSEC-06: Enforce data lineage

• MLSEC-07: Keep only relevant data

Reliability pillar

• MLREL-03: Use a data catalog

• MLREL-04: Use a data pipeline

• MLREL-05: Automate managing data changes

Performance efficiency pillar

• MLPER-04: Use a modern data architecture

Cost optimization pillar

• MLCOST-05: Use managed data labeling

• MLCOST-06: Use data wrangler tools for interactive analysis

• MLCOST-07: Use managed data processing capabilities

• MLCOST-08: Enable feature reusability

Sustainability pillar

• MLSUS-04: Minimize idle resources 

• MLSUS-05: Implement data lifecycle policies aligned with your sustainability goals

• MLSUS-06: Adopt sustainable storage options

Model development phase

Operational excellence pillar

• MLOE-12: Automate operations through MLOps and CI/CD

• MLOE-13: Establish reliable packaging patterns to access approved public libraries

Security pillar

• MLSEC-08: Secure governed ML environment

• MLSEC-09: Secure inter-node cluster communications

• MLSEC-10: Protect against data poisoning threats

Reliability pillar

• MLREL-06: Enable CI/CD/CT automation with traceability

• MLREL-07: Ensure feature consistency across training and inference

• MLREL-08: Ensure model validation with relevant data

• MLREL-09: Establish data bias detection and mitigation

Performance efficiency pillar

• MLPER-05: Optimize training and inference instance types

• MLPER-06: Explore alternatives for performance improvement

• MLPER-07: Establish a model performance evaluation pipeline

• MLPER-08: Establish feature statistics

• MLPER-09: Perform a performance trade-off analysis

• MLPER-10: Detect performance issues when using transfer learning

Cost optimization pillar

• MLCOST-09: Select optimal computing instance size

• MLCOST-10: Use managed build environments

• MLCOST-11: Select local training for small scale experiments

• MLCOST-12: Select an optimal ML framework

• MLCOST-13: Use automated machine learning

• MLCOST-14: Use managed training capabilities

• MLCOST-15: Use distributed training

• MLCOST-16: Stop resources when not in use

• MLCOST-17: Start training with small datasets

• MLCOST-18: Use warm-start and checkpointing hyperparameter tuning

• MLCOST-19: Use hyperparameter optimization technologies

• MLCOST-20 - Setup budget and use resource tagging to track costs

• MLCOST-21: Enable data and compute proximity

• MLCOST-22: Select optimal algorithms

• MLCOST-23: Enable debugging and logging

Sustainability pillar

• MLSUS-07: Define sustainable performance criteria

• MLSUS-08: Select energy-efficient algorithms

• MLSUS-09: Archive or delete unnecessary training artifacts

• MLSUS-10: Use efficient model tuning methods

Model deployment phase

Operational excellence pillar

• MLOE-14: Establish deployment environment metrics

Security pillar

• MLSEC-11: Protect against adversarial and malicious activities

Reliability pillar

• MLREL-10: Automate endpoint changes through a pipeline

• MLREL-11: Use an appropriate deployment and testing strategy

Performance efficiency pillar

• MLPER-11: Evaluate cloud versus edge options for machine learning deployment

• MLPER-12: Choose an optimal deployment option in the cloud

Cost optimization pillar

• MLCOST-24: Use appropriate deployment option

• MLCOST-25: Explore cost effective hardware options

• MLCOST-26: Right-size the model hosting instance fleet

Sustainability pillar

• MLSUS-11: Align SLAs with sustainability goals

• MLSUS-12: Use efficient silicon

• MLSUS-13: Optimize models for inference

• MLSUS-14: Deploy multiple models behind a single endpoint

Model monitoring phase

Operational excellence pillar

• MLOE-15: Enable model observability and tracking

• MLOE-16: Synchronize architecture and configuration, and check for skew across environments

Security pillar

• MLSEC-12: Restrict access to intended legitimate consumers

• MLSEC-13: Monitor human interactions with data for anomalous activity

Reliability pillar

• MLREL-12: Allow automatic scaling of the model endpoint

• MLREL-13: Ensure a recoverable endpoint with a managed version control strategy

Performance efficiency pillar

• MLPER-13: Evaluate model explainability

• MLPER-14: Evaluate data drift

• MLPER-15: Monitor, detect, and handle model performance degradation

• MLPER-16: Establish an automated re-training framework

• MLPER-17: Review for updated data/features for retraining

• MLPER-18: Include human-in-the-loop monitoring

Cost optimization pillar

• MLCOST-27: Monitor usage and cost by ML activity

• MLCOST-28: Monitor Return on Investment for ML models

• MLCOST-29: Monitor endpoint usage and right-size the instance fleet

Sustainability pillar

• MLSUS-15: Measure material efficiency

• MLSUS-16: Retrain only when necessary