MLPER-01: Determine key performance indicators
Use guidance from business stakeholders to capture key performance indicators (KPIs) relevant to the business use case. The KPIs should be directly linked to business value to guide acceptable model performance. Consider that machine learning inferences are probabilistic and will not provide exact results. Identify a minimum acceptable accuracy and maximum acceptable error in the KPIs. This helps enable achieving the required business value and manage the risk of variable results.
Implementation plan
-
Quantify the value of machine learning for the business - Consider measures of how machine learning and automation will impact the business:
-
How much will machine learning reduce costs?
-
How many more users will be reached by increasing scale?
-
How much time will the business save by being able to respond faster to changes, such as in demand and supply disruptions?
-
How many hours of manual effort will be eliminated by automating with machine learning?
-
How much will machine learning be able to change user behavior, such as reducing churn?
-
-
Evaluate risks and the tolerance for error -Quantify the impact of machine learning on the business. Rank order the value of impacts to identify the primary KPIs to optimize with machine learning. Define the cost of error for automated inferences that will be performed by ML models in the use case. Determine the tolerance of the business for error. For example, determine how far off a cost reduction estimate would have to be to negatively impact the business goals. Finally, evaluate the risks of machine learning for the business, and whether the benefits of ML solutions are of high enough value to outweigh those risks.
Documents
Blogs
