You’ve built and launched your new agent. Fantastic!
Users are starting to interact with it. Even better!
You're tracking user clicks before, while, and after interacting with the agent. Okay, good start.
But what exactly did your users ask your agent? Were they satisfied with the response? Is your feedback limited to just thumbs up or down? How can you truly measure your agent's success, gauge ROI, understand what users are asking it to do that is missing or of low quality, and confidently plan your roadmap?
Today, enterprises are building and deploying agents without much strategic oversight or goals. The push towards AI innovation is outpacing our ability to measure, iterate, and optimize what’s currently working.
Let's explore four common misconceptions around agents, plus practical strategies for building powerful agents.
Misconception 1: “Start with the low-hanging fruit.”
While it might seem like a good idea to tackle simple use cases first, these are often too trivial and aren’t worth the resource investments. For instance, creating an agent that confirms a software user’s account details (like username or email) might be easy, but this information is already easy to find. The value of this agent is low, and the resources you invested in building it could’ve been spent elsewhere.
Instead, focus on meaningful use cases your customers genuinely care about, like automating tasks that save hours of manual work.
At Pendo, our first agent use case was in Pendo Listen's Feedback Agent, Explore, letting customers ask questions about their textual feedback data from multiple sources. This provided clear, immediate value by significantly reducing manual data analysis time and freed up our users for higher-value work.
Misconception 2: “Agent user problems are new, and hard to understand.”
User frustrations with agents mirror familiar issues from traditional software. In traditional software interfaces, sometimes users struggle to find the page or button of the action they’re interested in.
In the agentic world, this corresponds to a situation where users repeatedly try to rephrase their question clearly to an agent. The underlying user frustration remains the same. An error message in the UI corresponds with an “I cannot answer this question” response from the agent. User problems remain similar, they just have a different form when it comes to agents.
Moreover, agent interactions are actually making it easier for us to identify and solve frustrations! Because users tell agents exactly what they need in natural language, like “I want to transfer $100 from my account to my savings account,” it's far easier to understand the user intent and spot underlying performance issues.
Recognizing this early can help your team apply existing user experience (UX) knowledge to deliver smooth agent interactions.
If you have Pendo Agent Analytics, you can also understand patterns at scale. Maybe your agent excels in some tasks, like answering common how-to questions, but struggles with others, like handling detailed, technical troubleshooting questions. These insights help you improve roll-out strategies and plan your roadmap, all to guide investments and prioritize the use cases that your agent should support.
Misconception 3: “I shouldn’t measure agent performance until it's deployed and available to all customers.”
Waiting too long to evaluate your agent’s performance can lead to missed critical early insights. For example, you might discover during early testing that users frequently ask the agent about integration setup, even though that wasn't part of the initial scope. Or, you might see that users consistently abandon the interaction after a certain point, highlighting a UX or trust issue that needs immediate attention. You can surface valuable feedback quickly, even in initial testing stages with design partners.
Early-stage analytics—like Pendo Agent Analytics—can help you pinpoint user interactions, highlight limitations, and guide improvements long before a wide release. This early visibility allows your team to fix friction points before they scale and helps you avoid launching features that may not meet real user needs. It also helps you build trust faster, showing users that the agent learns from and improves with every interaction.
Misconception 4: “Agents will completely replace the traditional UI.”
Not everything is easier with a chat. Sometimes it’s just faster to click a button or scan a dashboard. Other times, users want to quickly complete a familiar task without conversation, like fixing a piece of code or updating a billing address. The most successful products blend traditional UI elements (like pages, buttons, charts, and links) with conversational agent experiences.
We’re already seeing this today:
- Claude Code and Bolt.new let users seamlessly switch between conversational interactions and the coding interface.
- Google Docs’ AI agent helps users phrase and embed content, while its traditional interface elements allow you to format fonts and draw tables.
The world isn’t black or white, and it doesn’t have to be agent-only or UI-only. Your product experience can and should include both. When planning your UX, think about how to blend the best of both world: where the agent can offer guidance or assist with decisions, and where traditional UI remains the fastest, clearest path for users to act.
Building your data-driven Agent roadmap
To continuously improve your agent and ensure it’s driving business value, closely observe user interactions to identify gaps, like questions or workflows your agent currently struggles with.
Pendo Agent Analytics not only reveals these gaps clearly but also integrates downstream feature usage data and page interactions, giving you a holistic view of user behavior patterns. You'll understand precisely where your agent excels and where additional focus is required. Starting analytics early means you'll shape your agent's evolution right from the outset—so why not build your agent with us?
