MLPER-01: Determine key performance indicators, including acceptable errors
Use guidance from business stakeholders to capture key performance indicators (KPIs) relevant to the business use case. The KPIs should be directly linked to business value to guide acceptable model performance. Consider that machine learning inferences are probabilistic and will not provide exact results. Identify a minimum acceptable accuracy and maximum acceptable error in the KPIs. This will enable achieving the required business value and manage the risk of variable results.
Implementation plan
-
Quantify the value of machine learning for the business — Consider measures of how machine learning and automation will impact the business:
-
How much will machine learning reduce costs?
-
How many more users will be reached by increasing scale?
-
How much more quickly will the business be able to respond to changes such as demand changes or supply disruptions?
-
How many hours of manual effort will be reduced by automating with machine learning?
-
How much will machine learning be able to change user behavior such as reducing churn?
-
-
Evaluate risks and the tolerance for error — Quantify the impact of machine learning on the business. Rank order the value of impacts to identify the primary KPIs to optimize with machine learning. Define the cost of error for automated inferences that will be performed by ML models in the use case. Determine the tolerance of the business for error. For example, determine how far off a cost reduction estimate would have to be to negatively impact the business goals. Finally, evaluate risks of machine learning for the business, and whether the benefits of ML solutions are of high enough value to outweigh the risks.
Documents
Blogs
Videos
