Introducing Conversions and Experiments in Pendo: How to make your in-app guides count

Today, the best product experience wins. It beats better but harder to use technology. It beats big ad campaigns. It beats time in market. It beats an army of salespeople. In the long run, there is no bigger advantage than having a product that people love and tell their friends about. It’s why more and more companies are becoming product led. 

One way to quickly improve your product experience is with in-app guides. With guides, you have a no-code way to quickly implement personalized onboarding, feature adoption campaigns, and much more. Not to mention in-app guides are contextual and often have a 100% higher conversion rate than email. 

But here’s the thing: The line between a guide being helpful and a guide being annoying is razor thin. And as more companies use guides, we’re seeing them cross that line into annoyance more often. At Pendo, we want to help make sure our customers’ guides improve the product experience, not detract from it. This is the spirit behind our latest release, guide experiments–a new way to test, measure, and prove the impact of your in-app guides in Pendo. 

Guide experiments is live today–if you’re a Pendo customer, click here to get started.

Gut-feel vs. strategic guides

Like any communication channel, guides that are used poorly become a distraction to users trying to accomplish their goal. We’ve all seen them. The confusing 20-step onboarding guide that you can’t get away from. The massive (and interruptive) product announcement. Or worse: guides that just cover up a product issue that never gets fixed.

No one sets out to purposely do this with guides, but they can get away from you easily. There are often many issues, like lack of engineering support or too many people creating guides, but what we see the most is a lack of strategy and measurement.

“[Guide experiments are] great because we’re actually able to test the impact of things. So now when someone wants to run a guide, my first question is, ‘Do you want to test it?’ as opposed to just launching it based on gut feel.” – Amber Britton, Senior PM at SecurityScorecard (Beta user)

Most bad guides are deployed based on gut feel, intuition, and quick hypothesis. This can work in some instances (if you’re lucky), but sooner or later fails. Guides are part of your product, and it’s important to take a strategic approach to them, too. Teams should implement guides based on data, have a clear goal, and make sure they improve your product experience, not make it worse.

Experiments and conversions in Pendo

We’ve learned several strategies from our customers for ensuring guides are helpful, not harmful. These teams build guide standards and centers of excellence. They take a very strategic, data-driven approach to creating guides. They implement a guide review process. But the Pendo product can help solve for this, too, and experiments is that solution.

“I’ve been running an in-app upgrade campaign for months and just guesstimating the impact it’s had on driving users to review pricing. Now–with experiments–I can say 143% more traffic went to the plans page when they see the guide, vs. not seeing it. That’s a huge win for me in my role that I can take to my boss.” – Amber Britton, Senior PM at SecurityScorecard (Beta user)

With the release of guide experiments, customers will see three new features: conversions, experiments, and a results dashboard. Here’s an overview of each of these new capabilities:

1. Conversions: Set and track goals for your guides

Note: this is available to all Pendo users

You can now set a conversion that defines and measures the goal of your guide. Since strategic guides have clear goals, conversions are how you’ll know if your Pendo guides are achieving that goal. 

Conversions track the number of users who engaged with the conversion target after seeing the guide. The conversion target can be the feature, page, or track event you expect users to engage with after viewing the guide–this is the guide’s success criteria. Conversions can be set up independent of an experiment, and are useful for collecting guide analytics without setting up an experiment.

2. Experiments: Did your guide influence user behavior? 

Note: this is exclusive to Pro and Enterprise Pendo users

An experiment creates a subset of users (control group), who won’t see a guide, allowing you to measure the impact your guides are making. You’ll be able to compare performance of the two groups, and clearly understand the behavior your guides are (or are not) driving.

Experiments tell you if a guide is helpful–for example, if the control group behavior is the same as people who see the guide, your guide isn’t doing all that much. This signals that the guide might just be in the way and should be changed or removed. On the flip side, experiments will also clearly display how many more people are converting because they saw the guide, which is a great way to show the impact of your work.

3. Results Dashboard: Measure guide results all in one place

This new dashboard brings everything together. Your experiments will start running automatically when a guide is published, and you can monitor the results from the experiment results tab. On this page, you will see a summary of the experiment results, a chart displaying conversions over time for the experimental and control groups, and a downloadable experiment data table.

The power of in-app guides

We’ll say it: We love guides. They can be a tremendously helpful tool for product teams and their customers, and we hope that our approach to Pendo Guides helps you build the best guides possible. Guides that are personalized using data and segmentation. Guides that are truly strategic and helpful, confirmed and improved using conversions and experiments. Ultimately, it’s up to you; it’s not easy, but your customers will thank you for taking a more strategic, data-driven approach to guides.