4 implications for PMs using the Confusion Matrix to set metrics so that performance objectives are clear for developers and stakeholders

True Positives

  • If you have a 100% true positive rate you predict correctly 100% of the time. I’m pretty sure my Mom’s advice model is believed to function at this rate.
  • e.g. If out of a sample of 100 new YouTubers 10 of them will hit 1MM subscribers, but the model predicts that 8 will, the true positive rate is 80%.
  • PM Implication: You want this rate to be high. If you were a PM on this product, your model would need to predict as close to 10 as possible and at a higher performance than the current system. The tricky part deciding what rate is good enough. You’ll need to calculate and decide at what true positive rate the investment into the AI/ML solution is worth it.
  • True Negatives
  • If you have a model that gives you 100% true negatives this means that if it says that a YouTube won’t make it, it’s right.
  • e.g. If out of 100 people, 90 YouTubers don’t make it and your model predicts that 85 will not, then it predicted that 5 too few would not make it and has roughly 94.5% true negative rate.
  • PM Implication: The rate needs to be at a level where people don’t lose trust in the model that tells them to invest coaching time in a YouTuber who’s going to get 3 subscribers and not 1MM.

False Positives

  • If a model gives you 100% false positives, any YouTuber who is predicted to reach 1MM subscribers actually won’t. This is a Type I error in statistics.
  • e.g. If out of 100 YouTubers, 10 actually reach 1MM subscribers and your model predicts that 15 will, it has a 50% error rate. So you should expect that a model will predict about 50% too many YouTubers to reach 1MM than there will be in reality. Note that in the 15 YouTuber prediction, 10 of those were true positives, it just went 5 YouTubers overboard.
  • PM Implication: False positive rates should be low as not to flag everything as worthy someone’s attention, even if they will catch all the winning YouTubers. The YouTube coaches may stop trusting the model. If everything is important, then nothing is.
  • False negatives:
  • If a model gives you 100% false negatives, any YouTuber who is predicted to not reach 1MM actually will. This is a Type II error in statistics.
  • e.g. If out of 100 YouTubers 90 don’t reach 1MM subscribers, but the model says that 95 will not then you have a 5.5% false negative rate and the coaches stand to not coach 5 star YouTubers.
  • PM Implication: You don’t want people to miss out on business, like not coaching valuable YouTubers. Therefore, this error rate must also be low.

Let’s explore further

AI Product Manager | Combing academic theory and product experience to create useful PM tools | Ultra-marathon runner

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

The Effect DAO Voting: Adopt to Survive

Role and Uses of NLP in Government

ML, DL, AI, Sounds confusing…?

The Lawyer’s Practical Guide: Legal Chatbots

How big businesses are using Machine Learning?

Where are the humans headed to?

“AI in manufacturing” resonating in the academia

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ilona Melnychuk

Ilona Melnychuk

AI Product Manager | Combing academic theory and product experience to create useful PM tools | Ultra-marathon runner

More from Medium

FFDW and Guardian Project Team Up to Bring Decentralized Storage to Content Verification and…

What is the BEST tool to improve speaking in your target language?

Internal Communications Introductions: Meet Toby Frankenstein

Bar Daily: CTO interview: James Donkin, Ocado Technology