User Experiments with In-App Guides? Make Sure You Know What to Measure

In-app guides provide a unique avenue to product and customer success teams–they open up the application itself as a communication channel with users. More importantly, the guides can be added directly to the user experience independently from any development cycles. This freedom opens up an enormous opportunity for experimentation.

Through user segments of both randomized samples and specifically designated groups, teams can design experiments to test different messages in the application. Does a tooltip increase feature usage? Are announcements more effective in a banner or lightbox? What wording drives a higher rate of action? What sequence or number of steps in a walk-through provides the best learning experience? Can we drive upsells with the right in-app nudge? These can all be easily tested using segments.

How Do You Know a Guide is Effective?

Excited? Great, you’re ready to start experimenting. Now, one important question before getting started. How will you measure the results? How do you know if a guide is successful? This key question is at the heart of why we built Pendo. It’s generally easy to measure the engagement level with a particular message. Did it get viewed? Did the user dismiss it, or click on an action within the guide? How many steps in a walk-through did the user complete before leaving? These are very useful measures, but they don’t really get at the reason you tested the guide in the first place–to try and change some aspect of user behavior.

To adequately measure the effectiveness of your guides, first, you need to understand the user behavior outcome that you want to effect–the more specific the better. Do you want to increase usage of a feature in your application? Increase the login frequency of a particular user segment? Have a higher completion rate for a specific task in the application? Be sure you can measure the before and after state. Only then can you actually judge the effectiveness of your guides.

Regular users of Pendo will know that we have made a number of changes to our guide metrics over the past few months. The goal of these changes is to make it easier for our users to capture these types of measurements and ensure that they can easily understand both engagement-level with guides as well as how their guides are influencing user behavior.

Learn Some Best Practices

Want to learn some more about how you can make the best use of guides in your application? As an end-to-end leasing platform for commercial real estate, View The Space, Inc. (VTS) has a number of user roles they need to reach including landlords, agency brokers, and tenant reps. Next week (July 27th at 1:00PM ET) Josiah Wang, a product manager at VTS, will join us for a special “Pendo Stories” webinar and share how they use in-app guides for better training and communication with their users. Watch the webinar today.