MLCOST-23: Enable debugging and logging

Ensure that there are sufficient logs and metrics recorded to capture the runtime and resource consumption. The collected logs and metrics can be analyzed to identify the areas for improvement. Monitor compute and data storage consumption. Instrument the machine learning code, and use debugging tools to capture metrics at runtime.

Implementation plan

• Use Amazon SageMaker Debugger - Amazon SageMaker Debugger captures the state of a training job at periodic intervals. It provides visibility into the ML training process by monitoring, recording, and analyzing data with the ability to perform interactive exploration of data captured during training. The debugger has an alerting capability for errors detected during training. For example, it can automatically detect and alert you to commonly occurring errors, such as gradient values getting too large or too small.

• Use Amazon CloudWatch -Logs generated during training by Amazon SageMaker are logged to Amazon CloudWatch Logs. Use an AWS KMS key to encrypt log data ingested by Amazon CloudWatch Logs.

Documents

• Logging and Monitoring

• Logging Amazon ML API Calls with AWS CloudTrail

• Amazon SageMaker Debugger - Setup and Use

Blogs

• Build Your Own Log Analytics Solution on AWS

• Profile Your Machine Learning Training Jobs with Amazon SageMaker Debugger

• Amazon SageMaker Debugger – Debug Your Machine Learning Models

• ML Explainability with Amazon SageMaker Debugger

• The science behind SageMaker’s cost-saving Debugger

Examples

• Debugger example notebooks

Video

• Train ML models faster with better insights using Amazon SageMaker Debugger

• Debugging a Customer Churn Model Using SageMaker Debugger