Day in the Life of an AI/ML Platform Product Manager: Where to Measure Impact with Numbers and where to tell Stories.

As a product manager of an AI/ML platform, there are two types of metrics I care about, first and second-order metrics. The direct impact that our team’s work has is measured by first-order metrics. The indirect impact that our team’s work will have is a contribution to second-order metrics. You may also know second-order metrics as ‘proxy’ metrics.

Here’s an example from my team:

First-order metrics:

As an ML platform team, we care about providing a golden path for data scientists to develop and deploy their ML models. Our North Star first-order metric is high data science speed. If the only concern that data scientists have is creativity in how to get value from data then we’re happy.

Second-order metrics:

Data Scientists build models to make predictions that will be used within a product and add value to end users. An example is predicting the likelihood of our company clients being fraudulent. Some of the metrics they care about are recall and precision rates. For me on the platform side, the recall and precision metric is a second-order metric.

3 reasons it’s important to align with second-order metrics:

So while having a perfect platform won’t have a direct impact on precision and recall (first-order metrics), it’s still useful to align to these metrics for three reasons; be able to prioritise what problems to solve; to show the value of the platform to the business, and to motivate engineers by showing their impact on business outcomes

Numbers and stories

The difference between how you work with these metrics is that you measure first-order metrics with numbers, and you use words and sentences to tell a story on how you align with second-order metrics.