How to become a data-driven product manager

From the responses, we can pull out a handful of key themes: using data to make decisions in service of your customers, leveraging data throughout the product life cycle, collecting data from multiple channels, and balancing hard data with your own instincts. We’ll dig into each of these in more detail throughout the rest of the e-book, but let’s start with the elephant in the room: the human side of data analysis.

• Using data at the right time
• Balancing signals
• Using the right amount of data
• Intuition
• Entire product life cycle
• Actionable data
• Addressing customer needs
• Understanding users
• Qualitative and quantitative
• Data-informed
• Connecting the organization

There is no data-driven product management without the PM

In his latest book, 21 Lessons for the 21st Century,, historian Yuval Noah Harari writes that “in a world deluged by irrelevant information, clarity is power.” For product managers, that irrelevant information (and hopefully, plenty of relevant information) is data, coming from a multitude of sources. It is then up to the PM to make sense of it.

As we heard in the previous section, “data-driven” doesn’t necessarily mean blindly following the data. In fact, the person tasked with analyzing the data in front of them is equally as important (if not more so) as the data itself. And there are two keys areas where the human element of data analysis can’t be swapped for an algorithm or automated system.

1. Making decisions

No matter how informative or eye-opening the data may be, there is always going to be a need for some sort of human judgement. According to Tatyana Mamut, head of product at Nextdoor, the job of a product leader is to interpret the data effectively, understand its meaning, and then move forward depending on whether the situation requires a decision based on empirical observation or values (or both).

Travis Turney, senior data strategist at Rapid7, echoed this sentiment: “You’re never going to have a program where you put in some inputs and it spits out exactly what the solution is. You can’t have data analysis without a human in the loop.”

Sure, there are some questions in product management that can be answered with a single data point—which feature has more usage? What’s our average NPS score? What percentage of X type of users did we retain after three months?—but there are many more that cannot be.

If you’re trying to figure out why users are dropping off at a certain point in the workflow, or how usage has changed for a particular feature, or what version of an in-app guide will drive the most action, there needs to be a human decision- maker involved.

2. Talking to customers

Everything a product manager does is in service of the customer, and the best way to understand what your customers need or want from your product is to talk to them directly. Greg Bayer, SVP of product at Nielsen, says that spending time with customers periodically to get feedback on specific “ features, help with roadmap planning, or even learn how they use complementary technologies “informs a macro point of view for a product manager that is critical for the full vision.”

Even if you think you know what customers want based on how they interact with (i.e. from product usage data) or feel about (i.e. from NPS or CSAT surveys) your product, nothing can replace the insights you gain from face-to-face (or virtual) conversations. Sam Benson said that for the product operations team at Firefly Learning, it’s extremely important to pair quantitative data with qualitative research from user interviews to understand the “why” at every stage of the discovery process.

Product managers must use data responsibly

The more data product managers use, (ideally) the more value it brings. But something Beatrice Fabris, manager, UX, content, and development at Mimecast, realized quickly is “ that you can often spin the data any way you want. Although quantitative data is thought to be objective, it can still be misused depending on how the human behind the analysis chooses to present it.

Product managers have a responsibility to use data in a way that doesn’t skew or hide the full picture. There will always be a need for product managers to use their own intuition, experiences, and opinions to make the right decisions. But data plays a big part in informing these decisions at every stage of the product development lifecycle.

How product managers use data

While data is always useful for a product manager, it’s useful in different ways. Rekha Venkatakrishnan, a senior manager of group product management, sees data as an ongoing part of the product development life cycle, but the key is to ensure you know the data you want to use (and why) to get to your goals.

Beatrice Fabris from Mimecast echoed this sentiment: “I wouldn’t actually say there’s one area where data is the most valuable. Data can be used throughout the whole process. Making sure we use the data in the right way is where we get the value from.”

She went on to explain that on the UX team, they approach data differently depending on the phase a product is in. Before a new feature release, she looks at where users spend a lot of time in the product to inform how they release complementary in-app guides. For a product that’s already out in the market, they use data to identify areas that need to be improved, and to benchmark in-app guide performance.

All that being said, it’s worth highlighting areas of the product development and customer life cycles where data can play an important role:

Discovery

In the discovery process, product managers can leverage data to identify any gaps in the product to inform what changes or updates to pursue. On the flip side, it’s also useful to look at your most used features to see if there are any ways to improve that experience for your users. If you know a particular feature is associated with key customer outcomes, you should always be looking for new ways to make it better.

ロードマップ

Product usage data is valuable for directly informing the product roadmap (i.e. which features are customers using the most, and therefore require more attention?), but Viraj Phanse, product manager at Amazon Web Services, brought up another scenario. He said that sometimes, a product manager will have an idea for the roadmap but no data available to back it up. In this case, PMs still need to make the call one way or the other and take smart risks. Adopting a customer- centric approach can help as well: focus on your customers’ problems throughout the roadmap journey, and constantly ask yourself what customer problem you’re solving.

Feature launches

For new feature launches, Greg Bayer of Nielsen says his team always tracks adoption and usage first and foremost, including: how many customers are using the new feature, and is it leading to more overall engagement in the product? He emphasized the importance of linking success criteria with your customers because at the end of the day, you want them to be successful because they used your product.

You should also analyze a new feature’s success to understand any usage patterns and what customers might still need from the product. Most importantly, make sure you have tracking in place before going into a new feature launch. Travis Turney of Rapid7 believes the metric itself doesn’t matter as much as having some sort of measurement program so you can learn over time what “good” looks like.

オンボーディング

When building an onboarding strategy, product data can help you determine which features to include in your onboarding flow (aka those that are most likely to lead to success in the product) as well as help you measure onboarding effectiveness.

Travis Turney of Rapid7 also called out onboarding as an important time to learn about your customers through the data you collect. Since customers are more likely to feel comfortable providing information about themselves when they first start using your product, Travis says it is your best opportunity to capture data (e.g. their role in the company) that can then allow you to provide a more personalized product experience going forward.

Experimentation and testing

The experimentation process hinges on data as you test, analyze, and learn what works and what doesn’t. Manosai Eerabathini of Google sees data as crucial to experimentation, especially for products that are well established and where testing and implementing small tweaks to the user experience can make a large impact.

Andy Browning, UX writer at Mimecast, explained this from the perspective of the UX team: “We found that when we do user testing, we have our own ideas and instincts going in, but then we actually see the users engaging with the product and get the data, and we might be completely wrong. And you can’t really argue with it—you know where you need to make improvements.”

リテンション

Retention is always important, but it has become even more critical for businesses to stay ahead of retention by understanding how customers are using the application. Sam Benson explained how at Firefly Learning, customer success managers (CSMs) need to have an idea of their customers’ user journeys in order to have productive conversations, especially when it comes time for renewals. This way, CSMs can offer the right recommendations tailored to each customer account’s usage.

You can learn the most from customers who don’t stick around

Data doesn’t start and end with the PM

When used effectively, product data exists as a shared language across the company. Nearly every team has a
reason to care about how the product is performing or how customers are using it, especially at organizations that are “ product led. So, a big part of a product manager’s job is to empower the rest of the organization with the data and insights they’ve collected.

Here are some tips for sharing product data effectively:

• Ensure all stakeholders have access to product data
• When sharing product data, deliver the information that’s relevant to each type of stakeholder (e.g. different teams)
• If you have a north star metric, utilize that as your mechanism for sharing product data
• Include product data in what’s shared at company-wide all-hands meetings or town halls
• Provide context as to why the data is important to each stakeholder or team specifically
• Use Slack (or another communication channel) to share product data and foster collaborative conversations

A product ops use case

Metrics that matter

Using data to build a great product is only half the story. Product managers need to track things like product usage, user sentiment, and customer feedback in order to continuously provide the best product experience and ensure customers’ needs are met. Even Sweetgreen, a fast-casual salad chain, identifies as a technology company, and is shifting to tracking metrics like active users in their app and lifetime customer value rather than just store sales or foot traffic.

Humans naturally gravitate toward information that is organized and when possible, simplified. And one of the most difficult exercises can be determining which product metrics to measure and then, which metrics matter the most. When we asked our interviewees what the most important metrics are for PMs to measure, there was one overwhelming consensus: it all depends.

Here are some of the factors that impact which metrics product managers should track:

• Whether your company is B2B, B2C, or a hybrid
• Your industry (including Saas vs. non-Saas, enterprise
  vs. consumer)
• The stage your product is in
• Your organization’s goals (e.g. new customer acquisition or retention)

Since there is no one-size-fits-all for product metrics, here are some different ways to approach choosing what to focus on:

Two example metrics frameworks

1. Viraj Phanse of Amazon Web Services shared a metrics framework whereby you categorize metrics by the internal team or external stakeholder they are relevant to. Here are sample metrics for each of his four defined categories:

• Management: revenue, profitability, Net Promoter Score (NPS), retention, customer lifetime value (CLV)
• Customer: NPS, customer satisfaction (CSAT) score, customer effort score (CES)
• Marketing: CLV, conversions, growth in daily and monthly active users
• Engineering: number of issues, velocity, click-through rate, NPS

2. Tatyana Mamut of Nextdoor shared the metrics framework her team developed, which is broken down into four categories as well:

• Product value: these are lagging indicators of the product’s success, for example engagement metrics (e.g. DAU/ MAU) and revenue.
• Product love: these help the product team understand if they are not only building a product that customers use, but one that they love. Ideally, they are leading indicators like week 4 WAU and NPS.
• Product purpose: most companies have a purpose statement, but don’t necessarily measure their effectiveness at delivering on that purpose. At Nextdoor, the team does this through a Neighborhood Vitality metric, which they track by looking at the number of High Quality Connections that are made (online and offline) and the amount of reported content on their platform.
• Employee passion: since you can’t create a great product without passionate employees, Nextdoor measures employee passion (via frequent surveys and constant feedback) and elevates it to the top level of their measurement framework.

The north star metric

The idea of a north star metric is a common practice for product teams looking to narrow in on one data point. Since it can be challenging to balance all of the data coming in from multiple places, a north star metric helps teams focus. The only downside being that this could cause product managers to lose sight of other product signals that matter, too.

Manosai Eerabathini of Google wasn’t always a believer in the north star metric, but has come to recognize the value of collapsing your focus and being able to point to one overall metric to track progress. His approach to the north star metric is to build a hierarchy of supporting metrics that ladder up to your north star.

At Tray.io, the team created a custom north star metric called a user score, which aggregates multiple data points (e.g. daily active users, clicks, and other funnel metrics) into one high-level view. Bella Renney says she looks at this metric daily, usually by examining the number of users who are in three different thresholds of scores. This aggregated score also allows her team to drill into underlying data points to understand why a particular customer’s score has increased or decreased.

Her advice for deciding on a north star metric or compilation score of your own? “Take a long, hard look at what success is for your users and build your score around that specifically.”

Balancing quantitative and qualitative

Another theme that came up in nearly every conversation is that PMs need to balance quantitative and qualitative data. According to Greg Bayer of Nielsen, “a good product manager needs to blend both together equally.”

Here’s what that can look like:

At Mimecast, Beatrice Fabris says they use their product analytics tool to validate whether the problem areas the team hears most often in customer calls match where users are spending the most time in the product. The data complements qualitative information coming from support calls in order to form a more complete picture.

Travis Turney of Rapid7 said the power of quantitative and qualitative data was especially crucial when the team worked to optimize their onboarding flow. By combining information from support tickets and product usage, they were better able to identify areas of the product where they could invest more and provide a better experience for new customers.

Viraj Phanse of Amazon Web Services offered an example related to understanding customers’ problems—he believes product managers need to actually be on calls with customers in order to understand the problems they’re facing.

From there, it’s a matter of digging into the quantitative side of things by looking at NPS survey responses or cohort analysis for the particular feature where the customer is having trouble.
Next, we’ll hear what our experts believe product managers need in order to use data effectively.

How to be a better data-driven product manager

To help fellow product leaders continue improving, we asked our interviewees to name the one skill or tool that a data- driven product manager needs to do their job. Here’s what they had to say (PMs, take notes).