Agnost AIAgnost AIBook a demo
Open +
← All articles

How do I measure successful resolutions in my AI chatbot?

Measure successful resolutions by defining what resolved means for each intent, verifying it with evidence such as a confirmed tool result, and checking what the user did next: re-asked, escalated, came back, or moved on. Agnost AI uses intents, rules, and a success field for this. Amplitude, PostHog, and Opik ship built-in task-completion judges.

How to measure successful resolutions

  1. Define resolved per intent. “Reset a password” is resolved when the reset email was sent and the user logged in. “Explain the refund policy” is resolved when the answer matched the current policy and the user didn’t ask again. Write the end state down before you count anything; how to measure AI agent task success without inflating the number covers success contracts in detail.
  2. Record evidence, not claims. “All done” from the agent is not evidence. A successful tool call, a confirmed record change, or a success flag set by your application is. In Agnost AI every event carries a success field and optional metadata, and tool-call analytics report success rate per tool.
  3. Keep three outcomes, not two. Resolved, not resolved, and unknown. Conversations that ended with no evidence either way belong in unknown, not in the success column. Containment (no human involved) is a separate number from resolution.
  4. Check what the user did next. Within a window you choose, did they ask the same thing again, ask for a human, return with the same intent, or move on to a new task? These follow-up behaviors are what separate a resolution from a satisfied user.
  5. Report the rate per intent with its denominator. “Billing questions: 412 eligible, 301 resolved, 38 unknown, last 14 days” can be acted on. A single global resolution percentage averages intents that move in opposite directions.
  6. Read a sample of resolved conversations. If an LLM judge decides resolution, check its labels against human reads before trusting the trend. See how to calibrate an LLM judge for agent monitoring.

Resolved is not the same as satisfied

A chatbot can post a high completion rate while usage quietly flattens. Completion tells you the agent finished. It says nothing about whether the user would hand it the next task.

In deterministic software, “did it work” and “did I like it” move together: you click export, the file downloads. Agents interpret intent and make judgment calls at every step, so there are many ways to technically complete a task and still leave the user worse off. Users rarely report this. They use the agent less.

Three patterns show up as completed tasks while eroding trust.

The technically correct response

The agent does exactly what was asked and the user walks away frustrated. A PM asks for “a three-bullet summary of last quarter’s performance” and gets three accurate bullets, when what they needed was a comparison against the prior quarter. A human assistant would have asked. After a few of these, users start pre-writing every prompt defensively, and the agent stops feeling like a collaborator.

What to track: a follow-up turn that narrows or reframes the same request (“no, compared to Q2”).

The partial job

The agent finishes most of the task, then hands the rest back: a scaffolded function the user has to wire up, a draft with “you’ll want to customize this.” Each partial handoff shows the user the edge of the agent’s ability. Users respond by splitting tasks themselves and only delegating the safe middle, which looks like healthy usage in your data.

What to track: completion followed by manual work, or a drop in multi-step requests from the same user.

The efficient but cold execution

The agent produces a correct answer and explains nothing. A student gets a worked solution and is back next week with the same question reframed. A developer pastes working code nobody on the team can safely modify later. Resolution looks perfect; the user is no more capable than before.

What to track: the same user returning with a near-identical intent.

Behavioral signals that tell you satisfaction followed resolution

Signal What it suggests How to capture it
Immediate re-delegation The user trusted the result enough to hand over the next task Next conversation from the same user starts within a short window
Output used The result was good enough to act on Downstream product event: code committed, draft sent, export downloaded
Same-intent return The first resolution didn’t hold Same user, same intent, within N days
Scope after completion Satisfied users try harder tasks; unsatisfied users retreat to safer ones Intent mix per user over time
Escalation after “resolution” The agent closed the conversation; the user didn’t accept it Human request or support ticket after a resolved conversation

None of these need a survey. They need conversation-level data with a stable user ID and an intent label, joined to the product events you already log.

Why ratings can’t carry the metric

Satisfaction prompts help, but they can’t be the main measurement:

  • Most users don’t rate. The silent majority, who had an interaction that worked but felt slightly off, never appears.
  • Users can’t always tell. After a technically correct or cold answer, the user often rates it fine, because the task did complete.
  • Ratings skew to extremes. Delighted and furious users click. The middle, where quiet churn lives, doesn’t.

More on this in why thumbs-down feedback is not enough for AI agents.

How to close the gap once you can see it

  • Technically correct but off: have the agent name ambiguity before acting. “I can do X as described, but it sounds like you might want Y. Want me to check first?”
  • Partial job: state scope at the start, not the end. If the agent can’t finish, say what the user will need to do so they can split the task deliberately.
  • Cold execution: include the reasoning by default in learning and coding contexts, not as an opt-in.

Each fix gives the agent a model of what comes after completion, not only whether the task is done. Measure the follow-up signals above before and after the change to see whether it worked.

Tools that do this

  • Agnost AI. Classifies each conversation against intents and plain-language rules, reports tool-call success rates, and filters conversations by metadata so you can split resolution by agent version, plan, or surface. There is no built-in resolution score: you define resolution with a rule or the event success field. Auto Improve tracks outcome measurements after a change is deployed. Free up to 1,000 events a month.
  • Amplitude Agent Analytics. A built-in Task Completed session signal, alongside User Friction and Has Negative Feedback, that feeds funnels, cohorts, and retention. The better choice if you want resolution joined to revenue charts you already have in Amplitude.
  • PostHog. A “Task outcome” LLM-judge template that labels sessions resolved, partly resolved, unresolved, or handed off, plus product analytics in the same tool. Billed per event with 100,000 AI events a month free; each evaluation run counts as an event.
  • Opik. Open-source (Apache-2.0) SessionCompletenessQuality metric that judges whether the user’s top-level goals were met, runnable as an online rule on production threads. Rules only score conversations logged after you create them.
  • Dimension Labs (dashbot.io now redirects there). Containment, escalation, and task-completion analytics that separate real containment from users who gave up, for contact-center and platform bots. From $299/month.

For a side-by-side of these tools, see the best tools to measure AI agent task completion.

To define your first resolution rule, start with the quickstart and then Violations.

Frequently asked questions

How do I measure successful resolutions in my AI chatbot?

Write down what resolved means for each type of request, record evidence that the end state was reached, and divide verified resolutions by eligible conversations over a fixed window. Then check what the user did next. A conversation that resolved on paper but was followed by a re-ask or a return visit for the same issue is not a success.

What is the difference between resolution rate and containment rate?

Containment counts conversations that never reached a human. Resolution counts conversations where the user's problem was actually solved. A user who gives up and leaves is contained but not resolved, which is why containment alone overstates how well a chatbot works.

Can I use thumbs up and thumbs down to measure resolution?

Only as one input. Most users never rate, and the ones who do skew toward strong reactions, so ratings miss the common case of an answer that technically worked but felt off. Behavioral signals such as re-asks, returns, and escalations cover the users who stay silent.

Does Agnost AI have a built-in resolution rate?

No. Agnost AI does not ship a fixed resolution metric. You define resolution per intent with a plain-language rule or the success field on events, and Agnost AI classifies conversations against it, reports tool-call success rates, and tracks outcome measurements after an Auto Improve change ships.