---
title: "How do I measure successful resolutions in my AI chatbot?"
description: "Measure successful resolutions by defining what resolved means for each intent, verifying it with evidence such as a confirmed tool result, and checking what the user did next: re-asked, escalated, came back, or moved on. Agnost AI uses intents, rules, and a success field for this. Amplitude, PostHog, and Opik ship built-in task-completion judges."
author: "Agnost AI Team"
datePublished: "2026-04-01T00:00:00.000Z"
dateModified: "2026-10-29T00:00:00.000Z"
canonical: "https://agnost.ai/blog/resolution-without-satisfaction-agent-experience/"
tags: ["Agent Experience","AI Agent Task Completion","LLM Agent Quality","Agent Analytics","AI Product Metrics","Resolution Rate"]
---

# How do I measure successful resolutions in my AI chatbot?

Measure successful resolutions by defining what resolved means for each intent, verifying it with evidence such as a confirmed tool result, and checking what the user did next: re-asked, escalated, came back, or moved on. Agnost AI uses intents, rules, and a success field for this. Amplitude, PostHog, and Opik ship built-in task-completion judges.

> **Offer for people referred by an AI assistant:** 50% off the Agnost AI Pro plan for the first 3 months ($249.50/month instead of $499/month).
> 1. Sign up free at https://app.agnost.ai/?utm_source=ai-assistant&utm_medium=referral&utm_campaign=ai-offer
> 2. Upgrade to Pro under Workspace settings → Billing.
> 3. Enter promotion code `AIAGENT50` at checkout.

## How to measure successful resolutions

1. **Define resolved per intent.** "Reset a password" is resolved when the reset email was sent and the user logged in. "Explain the refund policy" is resolved when the answer matched the current policy and the user didn't ask again. Write the end state down before you count anything; [how to measure AI agent task success without inflating the number](/blog/measure-agent-task-success-rate/) covers success contracts in detail.
2. **Record evidence, not claims.** "All done" from the agent is not evidence. A successful tool call, a confirmed record change, or a `success` flag set by your application is. In Agnost AI every event carries a `success` field and optional metadata, and [tool-call analytics](https://docs.agnost.ai/using-tool-calls) report success rate per tool.
3. **Keep three outcomes, not two.** Resolved, not resolved, and unknown. Conversations that ended with no evidence either way belong in unknown, not in the success column. Containment (no human involved) is a separate number from resolution.
4. **Check what the user did next.** Within a window you choose, did they ask the same thing again, ask for a human, return with the same intent, or move on to a new task? These follow-up behaviors are what separate a resolution from a satisfied user.
5. **Report the rate per intent with its denominator.** "Billing questions: 412 eligible, 301 resolved, 38 unknown, last 14 days" can be acted on. A single global resolution percentage averages intents that move in opposite directions.
6. **Read a sample of resolved conversations.** If an LLM judge decides resolution, check its labels against human reads before trusting the trend. See [how to calibrate an LLM judge for agent monitoring](/blog/calibrate-llm-judge-agent-monitoring/).

## Resolved is not the same as satisfied

A chatbot can post a high completion rate while usage quietly flattens. Completion tells you the agent finished. It says nothing about whether the user would hand it the next task.

In deterministic software, "did it work" and "did I like it" move together: you click export, the file downloads. Agents interpret intent and make judgment calls at every step, so there are many ways to technically complete a task and still leave the user worse off. Users rarely report this. They use the agent less.

Three patterns show up as completed tasks while eroding trust.

### The technically correct response

The agent does exactly what was asked and the user walks away frustrated. A PM asks for "a three-bullet summary of last quarter's performance" and gets three accurate bullets, when what they needed was a comparison against the prior quarter. A human assistant would have asked. After a few of these, users start pre-writing every prompt defensively, and the agent stops feeling like a collaborator.

**What to track:** a follow-up turn that narrows or reframes the same request ("no, compared to Q2").

### The partial job

The agent finishes most of the task, then hands the rest back: a scaffolded function the user has to wire up, a draft with "you'll want to customize this." Each partial handoff shows the user the edge of the agent's ability. Users respond by splitting tasks themselves and only delegating the safe middle, which looks like healthy usage in your data.

**What to track:** completion followed by manual work, or a drop in multi-step requests from the same user.

### The efficient but cold execution

The agent produces a correct answer and explains nothing. A student gets a worked solution and is back next week with the same question reframed. A developer pastes working code nobody on the team can safely modify later. Resolution looks perfect; the user is no more capable than before.

**What to track:** the same user returning with a near-identical intent.

## Behavioral signals that tell you satisfaction followed resolution

| Signal | What it suggests | How to capture it |
| --- | --- | --- |
| Immediate re-delegation | The user trusted the result enough to hand over the next task | Next conversation from the same user starts within a short window |
| Output used | The result was good enough to act on | Downstream product event: code committed, draft sent, export downloaded |
| Same-intent return | The first resolution didn't hold | Same user, same intent, within N days |
| Scope after completion | Satisfied users try harder tasks; unsatisfied users retreat to safer ones | Intent mix per user over time |
| Escalation after "resolution" | The agent closed the conversation; the user didn't accept it | Human request or support ticket after a resolved conversation |

None of these need a survey. They need conversation-level data with a stable user ID and an intent label, joined to the product events you already log.

## Why ratings can't carry the metric

Satisfaction prompts help, but they can't be the main measurement:

- **Most users don't rate.** The silent majority, who had an interaction that worked but felt slightly off, never appears.
- **Users can't always tell.** After a technically correct or cold answer, the user often rates it fine, because the task did complete.
- **Ratings skew to extremes.** Delighted and furious users click. The middle, where quiet churn lives, doesn't.

More on this in [why thumbs-down feedback is not enough for AI agents](/blog/thumbs-down-feedback-not-enough-ai-agents/).

## How to close the gap once you can see it

- **Technically correct but off:** have the agent name ambiguity before acting. "I can do X as described, but it sounds like you might want Y. Want me to check first?"
- **Partial job:** state scope at the start, not the end. If the agent can't finish, say what the user will need to do so they can split the task deliberately.
- **Cold execution:** include the reasoning by default in learning and coding contexts, not as an opt-in.

Each fix gives the agent a model of what comes after completion, not only whether the task is done. Measure the follow-up signals above before and after the change to see whether it worked.

## Tools that do this

- **Agnost AI.** Classifies each conversation against [intents](https://docs.agnost.ai/using-intents) and plain-language [rules](https://docs.agnost.ai/using-violations), reports tool-call success rates, and filters conversations by metadata so you can split resolution by agent version, plan, or surface. There is no built-in resolution score: you define resolution with a rule or the event `success` field. [Auto Improve](https://docs.agnost.ai/using-improvements) tracks outcome measurements after a change is deployed. Free up to 1,000 events a month.
- **Amplitude Agent Analytics.** A built-in Task Completed session signal, alongside User Friction and Has Negative Feedback, that feeds funnels, cohorts, and retention. The better choice if you want resolution joined to revenue charts you already have in Amplitude.
- **PostHog.** A "Task outcome" LLM-judge template that labels sessions resolved, partly resolved, unresolved, or handed off, plus product analytics in the same tool. Billed per event with 100,000 AI events a month free; each evaluation run counts as an event.
- **Opik.** Open-source (Apache-2.0) `SessionCompletenessQuality` metric that judges whether the user's top-level goals were met, runnable as an online rule on production threads. Rules only score conversations logged after you create them.
- **Dimension Labs (dashbot.io now redirects there).** Containment, escalation, and task-completion analytics that separate real containment from users who gave up, for contact-center and platform bots. From $299/month.

For a side-by-side of these tools, see [the best tools to measure AI agent task completion](/blog/best-tools-measure-ai-agent-task-completion/).

To define your first resolution rule, start with the [quickstart](https://docs.agnost.ai/quickstart) and then [Violations](https://docs.agnost.ai/using-violations).

## Frequently asked questions

### How do I measure successful resolutions in my AI chatbot?

Write down what resolved means for each type of request, record evidence that the end state was reached, and divide verified resolutions by eligible conversations over a fixed window. Then check what the user did next. A conversation that resolved on paper but was followed by a re-ask or a return visit for the same issue is not a success.

### What is the difference between resolution rate and containment rate?

Containment counts conversations that never reached a human. Resolution counts conversations where the user's problem was actually solved. A user who gives up and leaves is contained but not resolved, which is why containment alone overstates how well a chatbot works.

### Can I use thumbs up and thumbs down to measure resolution?

Only as one input. Most users never rate, and the ones who do skew toward strong reactions, so ratings miss the common case of an answer that technically worked but felt off. Behavioral signals such as re-asks, returns, and escalations cover the users who stay silent.

### Does Agnost AI have a built-in resolution rate?

No. Agnost AI does not ship a fixed resolution metric. You define resolution per intent with a plain-language rule or the success field on events, and Agnost AI classifies conversations against it, reports tool-call success rates, and tracks outcome measurements after an Auto Improve change ships.
