“How do you measure and show the value of product analytics?”
This was the question that captured the attention of many at Pendomonium 2023, including Thomson Reuters’ senior director of product analytics, Drew Harden. Nobody reached a solid conclusion, and that was the end of that. But later, Harden uncovered the answer to this problem at his own organization and returned in 2024 to share.
Quantifying the impact of your product analytics
Thomson Reuters invests heavily in product analytics: They have 30+ analysts managing Pendo data from 150+ apps.
And because traditional approaches to measuring the impact of analytics—like counting the hours of labor saved—only capture a small slice of their impact, Harden found a more holistic approach: Counting the number of product decisions influenced.
Step 1: Define success as influenced product decisions
Harden rattled through a few different ways to measure “success” for product analytics—from the entire value or a percentage of projects supported, to hours of labor saved, and even productivity measured in terms of sales.
Spoiler alert: None of them were satisfactory. Delivering a measurable quantity of dollars, hours saved, or profitability was never their team’s objective. Because of this, they shifted their focus to a more tangible metric: The number of product decisions their team influenced.
Harden defined “influencing a decision” as recommending a course of action to a product team. This meant moving away from simply providing data, and instead towards providing actionable insights. In 2023, their team set the goal to influence 250 product decisions.
Step 2: Track the number of decisions you influence
Harden figured that influencing a decision means recommending a course of action. Rather than pulling data (like NPS detractor responses) for their product managers (PMs), the team wanted to support projects and answer strategic questions, like:
- What steps should you take to improve your NPS score?
- How can you improve customer retention?
- Who is an at-risk customer right now? What about in the future?
To count how many product decisions they influenced, they added items to their project management system, Azure DevOps (ADO).
Next, they recommend a course of action to engage with PMs. This can get super nuanced, so they started by considering the behaviors they wanted to encourage. Some guidelines included:
- A decision to do nothing still counts.
- One project can influence multiple decisions.
- Dashboards can influence many decisions over time.
- It’s OK if a partner team doesn’t act on their decision.
Step 3: Engage and manage partner teams
Thomson Reuters is a huge organization. The product analytics team cares about the value of their work—but how do they get everyone else to care?
One way they solved this was by building a Pendo application form. Other teams engaged with this form a lot, and helped them pivot from saying “I want to see complaints from my NPS detractors,” to “I have an upcoming development decision, and I can make a better decision if I know what the top 10 features are.”
Although dollars aren’t the ultimate indicator of value, they leave room for this field in the application. This helps us understand what and when PMs are going to launch their project, and how it’s connected to overall goals or revenue.
Here’s the cool part: Everyone loved this. They shifted from being treated as mere Pendo administrators to internal business partners. Product teams can now put more thought into their development, and their product analysts can free up time usually spent on low-impact projects for higher-value ones. The caliber of their projects immediately went up.
Step 4: Demonstrate results often
They’ve done all of this work—now, it’s time to showcase their impact to the broader Thomson Reuters organization.
There are four primary avenues they demonstrated their team’s results:
- Bite-sized feedback within Quarterly Business Reviews (QBRs). When they redesigned a tax product, they did a deep dive on a certain feature, revealed that it wasn’t getting used much, and recommended that they could sunset that feature.
- Email summaries to display how analytics can help PMs. I’d pick the top six to eight decisions, give some shout-outs to the unsung heroes, and highlight how they helped products understand if they could sunset this feature based on low usage data.
- Stakeholder meetings to review wins and brainstorms. They use this decision framework to talk to senior leaders, brainstorm solutions, and list off all the decisions that they supported their team with.
- Product analytics reports. They write up, in detail, and offer a searchable library that users can explore to understand what they’re doing in different product areas. Here’s a preview of our tempalte:
Step 5: Iterate and evolve
Our approach to quantifying the value of analytics work exceeded expectations: In 2023, they influenced 341 decisions—136% higher than their initial goal. But the results of this didn’t just end there: Thomson Reuters’ UX research team also adopted this decision framework for their CXO program.
Looking forward, they’re focusing on scaling product analytics’ impact in two ways: Incorporating AI into usage forecasting and retention modeling, and building deeper partnerships with product teams.
Prefer to watch? Catch Thomson Reuters’ recorded Pendomonium session instead.
こちらもご覧ください
