Have you ever been to see a movie on the recommendation of a friend or family member and wished you hadn’t? My mom raved about La La Land for months before I saw it. She went as far as to predict it would be my new favorite movie. I liked the lead actors in other roles, I even like musicals, but somewhere in the first 15 minutes of the movie, I thought to myself, this shouldn’t be anyone’s favorite movie. Maybe the problem wasn’t the movie, maybe it was my expectations. I went in expecting it to be my new favorite movie, and that’s a tough bar to clear.
I might not love the movie La La Land, but I love a cold root beer float. In my mind, it’s an incredibly tasty drink. It turns out many people, particularly those from outside the US, think root beer tastes like old cough syrup and can’t believe Americans pay to drink it. Why? It’s new, it not a taste they’re used it, if they had grown up drinking it, they’d likely feel different.
Our previous expectations and experience matter. They shape our opinions and likelihood to enjoy a particular product or hate it. This extends beyond drinks and movies.
Experience and Expectations
Take a minute to think about the people who use the products you craft. Think back to what they did before your solution existed. How did they solve the problem your product solves? The way they solved the problem before you came along likely influences what they think and feel about your product and how you solve their problem.
In the last four years, I’ve become a big fan of using Net Promoter Score (NPS) to measure user sentiment and understand how I can provide a better experience for my users. In particular, I’ve learned a lot about my users by slicing and dicing NPS results into different user segments. With this idea of previous experience in mind, I set out to conduct an experiment.
My hypothesis was that users’ experience in solving their problem before they used my product would predict the likelihood that they recommend my product. To what extent I didn’t know. I surveyed all of my users and asked them what software they personally used the most prior to using my product. I also provided an option for manual process so I didn’t leave any groups out of the survey.
Next, I looked at NPS results by segments, with each segment signifying a cohort that used the same software before mine. The results were fascinating. If you’re familiar with NPS, you know that the scale is -100 to 100. In this experiment, I saw as much as a 40 point difference between these segments, with a statistically significant sample size.
Making NPS Actionable
So my experiment gave me great data, but what next? How could I take this from gee-whiz information to strategy? How could I make it actionable? This is where the science of product management meets the art of product management.
To improve your NPS score, and in turn, improve customer loyalty, you need to understand why certain users are detractors. Once you understand why they are detractors, you need to understand how to change it, or if you can change it at all.
If a person used the same product before yours for 20 years, it may be very difficult to win them over– and will inevitably take time– even if your product is objectively better in every category. Even if their previous solution was ugly and clunky, it was their ugly and clunky solution, it was comfortable because it was known.
Without changing your product entirely or moving away from your vision, listen and find ways to give users a few creature comforts, a few things that feel like home. To them, their world has been turned upside down, but you can make it a little more welcoming. Be patient, and be the best host you can be.
If a certain cohort tends to have more detractors because your product is somehow inferior to the product they previously used, then you probably have more work to do. Before you rush off to your next roadmap session, pause and think about the information you’ve just learned and ask yourself: how can I make up for what they’ve lost?.
Are the detractor comments with this cohort merely personal taste acquired through previous experience, or are they material, objective criticisms? Do you, as the product expert believe that, given time, these users will actually grow to love your product? If you can, take a look at the data, and see if it changes over time. How long have these detractors been on your product? Maybe they just need time to get used to life in the new world. Or maybe they need training. Maybe they’ve had both, but they’re still unhappy, in which case you need to find a way to help them feel comfortable or accept that you will lose them. As a product person, you are uniquely qualified to distinguish between these and solve for x.
