When ChatGPT burst onto the scene, we all knew something fundamental was about to change in the software world. The horizon for that change wasn't ten or fifteen years away—it was happening right now. 

As Pendo’s Sr. Manager of Product Management, Kruti Carsane worked on numerous projects, but her latest has quickly become one of the most exciting: figuring out how to deliver the best analytics possible for customers implementing AI agents.

She shared what she learned about the development of Pendo Agent Analytics—and why everyone should question their AI investments.

Q: Let's start with the basics: What are AI agents, and why should organizations care?

AI agents mean different things to different organizations, but most are building either agentic controls or conversational interfaces where users type input and receive helpful output. At Pendo, we think about AI agents as interfaces that let you interact with large language model (LLM) capabilities. 

For decades, digitization meant taking physical, manual processes and figuring out how to recreate them in software. Now, we're entering an era where you can simply tell a conversational agent what you want done, and it will go off, complete those tasks, and return with results.

This is a fundamental change in how people interact with software. Instead of clicking through workflows, users are having conversations. Instead of navigating complex interfaces, they're expressing intent in natural language.

Q: How did your team come up with Agent Analytics? 

We realized there was a massive blind spot in the AI deployment strategy most companies were following. Organizations were rushing to build and deploy conversational AI without any way to measure whether these agents were actually making things easier for users.

Nobody's asking themselves, “How do you know if people are actually using these AI agents?”, and “How do you know if things are genuinely faster because people are using them?”. 

Traditional analytics tools can tell you about system performance: uptime, response times, conversation volumes. But they can't answer the questions that really matter:

  • Are users successfully accomplishing their goals through AI interactions?
  • What types of questions are people asking, and are they getting satisfactory answers?
  • Is the conversational approach actually faster than traditional workflows?
  • Are AI agents driving adoption and engagement, or creating frustration?

Q: How does Pendo Agent Analytics work? 

Pendo's Agent Analytics captures comprehensive information about how users interact with your AI agents to provide insights that go far beyond conversation logs.

Pendo Agent Analytics tracks the complete user journey:

  • The types of questions users are asking your agents
  • Whether there's success within the context of those questions
  • What users' intent was when they started the interaction
  • Whether they were able to accomplish their intended goals
  • How AI agent usage connects to broader product adoption patterns

Most importantly, we answer the fundamental question: Is it actually faster and better for users to accomplish tasks through conversation with an AI agent than through traditional clicks and workflows?

This helps you to make data-driven decisions about your AI investments rather than hoping for the best.

Q: What’s the long-term vision for Agent Analytics, beyond what you’re launching now? 

What we're releasing now is just the beginning. We have a much broader vision for Agent Analytics that builds on Pendo's decade of experience understanding user behavior.

Over the past ten years, we've developed capabilities like Paths, Funnels, and Session Replay to help organizations understand how users interact with traditional software interfaces. These same analytical principles become even more critical as workflows shift from clicks to conversations.

The next level of insight will help you understand:

  • Which use cases are best suited for AI agents versus traditional interfaces
  • How to optimize agent responses based on user success patterns
  • Where conversational workflows create the most value for your organization
  • How AI agent adoption affects overall product engagement and retention

Q: Why is it so critical to measure AI agent performance beyond basic metrics? 

If conversational interfaces are going to replace traditional workflows (and the evidence suggests they will), then you need visibility into whether this transformation is actually beneficial for your users and your business.

Agent Analytics provides that visibility by answering critical questions:

  • Are your AI agents improving user productivity or creating new friction points?
  • Which types of users get the most value from agent interactions?
  • How should you prioritize AI development based on actual usage patterns?
  • Where are the gaps between user intent and agent capability?

Q: How do you see this shift to AI agents changing the broader software industry?

We're at an inflection point in software development. In the same way mobile forced us to rethink user experience design, AI agents are forcing us to rethink how we measure and optimize user interactions.

At Pendo, we've always believed that understanding user behavior is the key to building better software. As interfaces evolve from clicks to conversations, that principle remains the same—but the tools and metrics need to evolve too.

Agent Analytics represents our commitment to helping organizations navigate this transformation successfully. Talk to an expert about Pendo Agent Analytics and how it can transform your approach to AI measurement and optimization.