Data-driven User Guidance & Guidance-driven User Analysis

USER DEMOGRAPHICS

Context that falls in the “demographics” bucket includes basic information about the user, who they are, their locale, etc. as well as any additional customer-level detail that may be available. Assuming that we’re looking at a logged-in user in a commercial application, that detail can be extensive. Demographic context can include:

• Paying or trial customer
• Plan level
• Annual recurring revenue (ARR) for that customer
• Role in the application
• Functional job role

HISTORICAL BEHAVIOR

“Historical” context includes all of the user’s past behavior in the application. Here we’re basically looking at what a particular user has done in the application from the perspective of both time and activity. Historical context can include:

• First login
• Average time in the application
• Login frequency
• Features used
• Task completion rate
• Number of support tickets filled

CURRENT TASK

This component of user context looks at the task that the user is engaged in a specific moment. This can include the specific page in the application that is being used, a specific feature, or even a step in a defined task.

センチメント

This context is an understanding of the user’s overall satisfaction with the application. This can be captured via feedback, Net Promoter Scores (NPS), or even a health score imported from a customer success solution from Gainsight.

When brought together all of this context provides a deep understanding of the user’s goals, their experience and/or skill, and their overall happiness with the product. All of this context can help to determine not only whether to offer them additional guidance, but what content to emphasize within the guide.

Targeting in-application guides based on user context

All of the captured user context provides an opportunity to deeply customize guidance that is delivered to the user. But how should we think through the targeting approach? One way is to start with the following questions. Let’s think through what we know about the user:

• Have they been using the application a long time or are they new?
• Based on functional role, are they more likely to be technically savvy?
• What features or functions do they have access to based on their user role or plan level?
• Have they used this particular feature before?
• Are they working on a task where users generally struggle in the application?

The answers to these questions can help to form the ways a guide could be targeted. Someone who has used a feature multiple times (successfully that is), probably doesn’t need to be prompted for any guidance at all. Someone who is technically savvy might need less guidance, or perhaps explanations of additional, more advanced options. If they do not have access to certain options, then those options could be left out of a shorter walkthrough. A user with low overall satisfaction or health score might need additional help across the board. There are a wide range of ways that user context could influence how guidance is presented, and it may take some experimentation to find the right balance. It’s important to produce tailored content, but the extent of the customization needs to be balanced against the time invested in customization and the ongoing maintenance of the content.

Pendo simplifies the process of targeting in-application guides through the use of user segments. Because Pendo captures all of the user actions in the application once it is installed, most of the user context is already available. Additional pieces of demographic information can be passed to Pendo directly or via tools like Salesforce and Gainsight. Segments can be defined using any combination of demographic, behavioral, or satisfaction attributes, and then each guide that’s deployed in the application can be targeted to a segment. When a user is in the application, Pendo will determine if they are an appropriate target for the guide, and only show it to those who meet the criteria. All of the display logic and segmentation rules are handled by Pendo, allowing companies to focus on the help content itself.

Why invest energy in targeting in-application guides?

The idea of delivering highly-targeted help is one the resonates with nearly every software user, but there is definitely a cost involved. It’s certainly easier and faster to make all of the help content follow a one-size-fits-most approach. But is the trade-off worth it? Well, one key factor is relevance. As discussed in the introduction of this paper, on average 70% of users won’t advance past the first step in a tutorial walkthrough. Users simply won’t engage with content, unless it’s clearly relevant and valuable to them.

Another important consideration is the length of the in-application content. If an in-application training experience shows all of the possible feature help to all segments of users that means that each individual guide or walkthrough must cover quite a bit more content. Predictably, however, completion rates degrade pretty rapidly as the number of steps in a guide increases. On average, by the time you get to the sixth step in a walkthrough only about 6.4% of the visitors who started on the first step will still be consuming the content. Considering that the ultimate objective is to help users become more proficient and realize more value from the application, presenting more content than they process in a session isn’t helpful.

It is also important to consider the “clutter” factor that can arise from in-application guidance. Negative or white space in a user experience is extremely important to discoverability, and proper use of white space within the user experience can increase user comprehension by 20%.3 The more cluttered the user experience, the less likely it is that users will be able to discover and learn features on their own. Every additional guide that is added (especially those that show automatically) adds clutter. There is an important balance point when the added “help” actually detracts from a user’s ability to understand and achieve proficiency in the application.

Targeting in-application guides isn’t just a matter of making content more relevant to users. It also helps to ensure that the user experience isn’t being degraded at all by the additional content. When guides are properly targeted, users only view content that matters to them, and only when it’s needed. This ensures a high level of engagement with the content which drives the actual user behaviors that increase user satisfaction and realized value.

Analyzing outcomes

Speaking of outcomes, thus far we’ve discussed the importance of targeting when it comes to in-application guidance, but there’s another way in which in the confluence of guidance and accumulated user context can be beneficial. That has to do with understanding and measuring outcomes. Let’s step back and reflect on why we would want to offer help at all. User guidance (regardless of form) is designed to help users become proficient with the key functions in an application. How do you know if a user is proficient? Proficiency, measured through user activity in the application, and overall volume of activity – as well as features used – are good indicators of user proficiency.

These measures are part of the historical “behavioral context” that we discussed earlier. Capturing these aspects of user behavior over time provides a platform to measure the effectiveness of any newly introduced guidance. With improved help, do users use the application more often? Do they use more (more features or components) of the application? Ultimately, this is the benchmark of effective help.

In the absence of this context, many organizations rely on actual engagement with guidance as a proxy for effectiveness. Clearly it’s important to measure whether the help content is viewed, either through external help center pageviews, or clicks / views of in-application guides. However, knowing that someone has viewed help content is not the same thing as knowing that someone has learned how to complete a task as a result of viewing the content. By analyzing behavioral data in conjunction with a user’s engagement with help content, companies can get a much better view of whether the guidance they are developing is having a positive impact on users or not. Given that unnecessary in-application guidance should be avoided, it’s important to continually assess the effectiveness of any content delivered in the application, and remove things that are not adding value.

Pendo allows organizations to easily monitor both of these metrics: engagement with in- application guidance, and user outcomes in the same interface. Feature usage can be overlaid on guide viewership which allows for easy visibility into the impact of any particular guidance. After adding some tooltip-based help to a feature, did that feature usage go up or down? Pendo tracks that information. In the same vein, Pendo allows user segments to be defined based on whether they have viewed a particular piece of in-application guidance. This allows user behavior to be compared across cohorts who have engaged with help content vs those that haven’t. This type of visibility can illuminate whether guidance is having a positive or negative overall impact on the customer experience.

³ White space isn’t just a UX fad, AIGA, 2015

Data-driven guidance elevates the customer experience

Truly “in-context” user guidance can have an incredibly powerful impact on the overall user experience. Knowing that most users will rarely consult documentation if at all, software providers who can’t deliver a better help experience are left to hope that users will be able to sort out their own usability problems. In the new area of recurring revenue, this isn’t a particularly safe bet. Moving guidance in-application and targeting it appropriately based on the user context ensures that help is relevant and available when needed, but also guards against introducing any additional usability issues through UX clutter. Capturing and measuring user context also provides a rich foundation of product data with which to benchmark the effectiveness of the content itself.

At Pendo, we understand the value and power that comes from leveraging accumulated user context to target and assess the guidance that is delivered in the application. Our platform is designed to capture as much user context as possible without any significant engineering overhead, and makes that context readily available to support product teams as they are building out help content. In the same integrated interface, Pendo allows actual product usage to be analyzed in conjunction with guide viewership. With Pendo, software companies can easily target their guidance, and measure the impact of their guidance on the user experience.