10 Ways to Measure User Onboarding Effectiveness

When we ran our user onboarding survey of product management and customer success professionals in the fall, most respondents indicated that their current onboarding process was not effectively meeting their objectives. We also found that the companies who were able to deliver against their onboarding objectives had not only higher customer satisfaction, but significantly higher growth than their competitors.

We also asked survey respondents to share their biggest onboarding challenge. Product complexity was cited as the biggest obstacle to onboarding success, however, many respondents indicated that lack of visibility into onboarding effectiveness was a big challenge as well. Part of the visibility challenge is that there are many aspects of onboarding that can be measured, and it’s not always clear which measurements are the most important.

To think through onboarding measurement, it’s helpful to break measures down into 3 specific tiers.

Tier 1: Onboarding Engagement

Most of the time when we’re discussing software user onboarding, we’re actually thinking about the initial user experience in the application and/or training content that is provided to users when they first log into an application. This is an incomplete view of user onboarding, but it’s a useful frame of reference for the first tier of measurement.

At this level, you are looking to measure the user engagement with the onboarding content itself. You want to look at how many users actually spend time with your onboarding content, how much of it gets completed, and how much time it takes users to get through it. A couple key measures to consider at this level are:

• Engagement rate (any usage) for onboarding content
• Completion rate of onboarding walkthroughs or courses
• Average time spent in onboarding content

These measures won’t necessarily tell you the effectiveness of your onboarding, but they will tell you something about the quality of the content and its relevance. If users don’t engage with onboarding content, then it’s likely not perceived to be helpful with what they’re trying to do. If they don’t complete it, it may be too long, or that your content isn’t focused enough.

Tier 2: Product Usage

The next level of user onboarding measurement looks at user behavior within the product itself, not just the onboarding walkthroughs or other content. Here you are looking to measure the impact that your onboarding is having on how users use the product – after they have engaged with the onboarding content. You want to look at how much of your product is used, adoption of specific features covered in the onboarding content, how often users log into your application, and how much time they spend in the product when they do. A couple of key measure to consider at this level are:

• Depth of use (% of product features) that are used by the average user
• Usage rate of specific features covered in the onboarding training
• Overall average time in the application or login frequency

These measurements can start to give you a more accurate picture of your onboarding effectiveness. Ultimately the goal of onboarding is to increase user proficiency, and drive the types of activities in the product that are associated with happy customers. The best way to see if you’re having an impact is to measure the actual user behavior in app.

For best results, you’ll want to segment your users by their onboarding engagement level. Look for behavioral differences between segments that have and have not gone through the onboarding content, those that complete the content, and those that don’t. Usage differences between segments can help to pinpoint the specific outcomes associated with your onboarding experience.

Tier 3: Business Outcomes

Measurements at this level are the most important, and unfortunately the ones that are most difficult to attribute to onboarding. Here you are looking at specific business and customer outcomes – things that ultimately drive the success of your product. How much cost is associated with acquiring and supporting customers? How satisfied are they? What is their retention rate? Key measurements to consider at this level are:

• Trial or freemium conversion rate
• Number of support tickets per customer
• Net Promoter Score or other customer satisfaction measure
• Customer and ARR churn rates

The challenge in looking at these measurements is that they are, of course, influenced by a lot of different factors not just how well your onboarding walkthoughs are building user proficiency. However, user segment can help you get more understanding from these measures. Segment your users by engagement with onboarding content, and by product usage.

Do customers that engage with onboarding content generate fewer support tickets? What about those that complete the content? Are there specific behaviors, or features that when used regularly drive higher conversion or satisfaction rates? By carefully segmenting your user base you can start to see the specific impact that onboarding is having on the overall product success.

For companies that are working on a new or revamped onboarding experience (like we did with our Orient Express project), it’s helpful to track before and after measurements for all of these metrics. That way you can track the bottom-line effectiveness of your initiatives, and begin to understand the specific ROI that is associated with improved user onboarding.

Bringing these three tiers of onboarding measurement can give you a clear and complete picture of the effectiveness of your user onboarding. With better visibility you can begin to optimize your own user onboarding experience, and realize some of the key benefits that we uncovered in our survey. Read the full survey results to see how you stack up.

These 10 measurements not only help you track onboarding, but they are also key health indicators for your overall product. Companies that measure more have greater visibility into their customers, the value their product delivers, and an early warning system into possible customer churn. Before investing in onboarding, make sure you are equipped to properly measure the results.