MLSUS-07: Define sustainable performance criteria

Make trade-offs between your model’s accuracy and its environmental impacts. When we focus only on the model’s accuracy, we “ignore the economic, environmental, or social cost of reaching the reported accuracy.” Because the relationship between model accuracy and complexity is at best logarithmic, training a model longer or looking for better hyperparameters only leads to a small increase in performance.

Implementation plan

• Establish sustainable performance criteria - Define performance criteria that support your sustainability goals while meeting your business requirements, but not exceeding them.

• Make trade-offs - Acceptable decreases in model performance can significantly reduce sustainability impacts of your models.

• Stop training early - In Automatic Model Tuning, early stopping stops the training jobs that a hyperparameter tuning job launches early when they are not improving significantly as measured by the objective metric. Similarly, SageMaker Debugger provides rules to automatically stop a training job as soon as it detects an issue (such as bug, job failing to converge...).

Documents

• Automatic Model Tuning with SageMaker - Stop Training Jobs Early

• Amazon SageMaker Debugger - Built-in Rules: LossNotDecreasing

Blogs

• Optimize AI/ML workloads for sustainability: Part 2, model development

• Amazon SageMaker Automatic Model Tuning now supports early stopping of training jobs

Metrics

• Track the metrics related to the resources provisioned for your training jobs (InstanceCount, InstanceType, and VolumeSizeInGB)

• Measure the efficient use of these resources (CPUUtilization, GPUUtilization, GPUMemoryUtilization, MemoryUtilization, and DiskUtilization) in the SageMaker Console, the CloudWatch Console or your SageMaker Debugger Profiling Report