Agnost AIAgnost AIBook a demo
Open +
← All articles

Why is my AI agent's success rate high but users are still unhappy?

An AI agent's success rate can be high while users are unhappy because it counts whether a task reached an end state, not whether it was the right task, took a reasonable path, or produced something users trust without rework. Agnost AI surfaces these gaps with intents and rules; Amplitude, PostHog, and Opik pair task completion with friction signals.

How to find out why users are unhappy despite a high success rate

  1. Check what your success rate counts. Many success flags record that the agent produced a final answer or that a tool returned without an error. Neither proves the user’s goal was met. See how to detect when an agent claims success without taking action.
  2. Read “successful” conversations that show friction. Filter for successes preceded by rephrasing, followed by corrections, or ending in a request for a human or a polite “ok thanks.” Read ten. The pattern is usually obvious once you see it in context.
  3. Compare the task completed with the task asked for. Write down what the user actually wanted in each sampled conversation and check whether the agent delivered that or an easier, adjacent version.
  4. Look at the path as well as the end. For each task type, compare the number of steps, tool calls, and tokens across conversations. A success that took three times the usual tool calls is a slow, expensive success the user felt.
  5. Look for rework and verification. Follow-ups like “can you double-check that,” users re-running the same task, or users editing most of the output all mean the result was a draft, not an outcome.
  6. Follow users over time. Are the people who delegated complex tasks last month now only handing the agent simple ones? That retreat shows up before churn does.
  7. Redefine success, then track friction next to it. Tie success to a verified outcome for each task type, and report rephrasing, human requests, and rework alongside the rate, segmented by user goal.

Four ways an agent succeeds while users lose patience

These are not edge cases. They are the normal ways a completion metric drifts away from the user experience.

1. The task was completed the hard way. The output is correct, but the agent made fourteen tool calls, looped on a subtask, and re-fetched data it already had. The user waited far longer than necessary, and your cost per task climbed. A lean three-step completion and a fourteen-step one look identical in a success rate.

2. The task was completed, but trust eroded. The answer was mostly right, with one wrong detail or an unrequested assumption. The user caught it. Next time they check every output line by line. The success rate doesn’t move, but the user has gone from delegating to supervising.

3. The easy version was completed and the real intent missed. A user asks for “a summary of the Q3 sales data” because they need to explain which regions underperformed before a meeting. The agent returns a descriptive summary. Task complete, intent missed. Every agent category has a version of this: the research agent that summarizes without synthesizing, the coding agent whose code compiles but ignores the codebase’s patterns, the scheduling agent that finds a slot but ignores travel time.

4. The output needed significant rework. When users rewrite most of what the agent produced, the success rate records a one. The user’s experience is “I got a starting point, not a result.” Counting that as a full success is how teams end up making decisions on data that flatters them.

For the support-specific version of this problem, see the support conversations your agent resolved but your customer hated.

What a success rate measures and what it doesn’t

A task completion rate measures one binary: did the agent produce an output that meets the stated criteria? It does not measure:

Not measured by success rate What to look at instead
Whether the user’s underlying goal was served Intent match: the goal the user stated vs. what the agent delivered
How long or how expensive the path was Steps, tool calls, and tokens per task type, especially the slow tail
Whether the output was used as-is Rework and edit follow-ups
Whether the user trusts the agent more or less Verification requests and delegation scope over time
How the user felt getting there Rephrasing, corrections, polite exits, requests for a human

Don’t drop the success rate. It’s the right sanity check for a new agent: if the agent fails half its tasks outright, fix that first. The problem is when it becomes the only number, because its blind spots are exactly the ones that decide whether users keep delegating.

Why evals don’t catch it

Most offline evals are automated completion checks: does the agent finish this class of task with output that meets these criteria? The eval suite goes green, and the team ships.

In production, the experience degrades in ways that never register as failures: more verification, narrower delegation, more rework, and eventually users doing the task themselves because the agent is more trouble than it’s worth. Evals measure what is easy to specify ahead of time. The felt experience only shows up in real conversations. More on that gap in why your agent fails in production even when evals pass.

Tools that do this

  • Agnost AI. Shows the gap rather than a single success score. Intents show what users were trying to do, plain-language rules such as “deliver what the user asked for, not a narrower version of it” are checked against every analyzed conversation with evidence attached, and tool-call analytics show success rate and latency per tool. There is no built-in conversation success metric: you define success with a rule or the success field on events, and Auto Improve tracks outcomes after a fix ships. Free up to 1,000 events a month.
  • Amplitude Agent Analytics. Built-in Task Completed and User Friction signals on the same sessions, so you can find completed sessions that still had friction, then see them in funnels and retention. Free up to 5,000 agent sessions a month.
  • PostHog. LLM-judge templates for Task outcome and User correction, plus automatic positive, neutral, or negative labels on user messages. Judges need your own model provider key, and each evaluation run counts as an event.
  • Opik. Open-source (Apache-2.0) conversation-level judges, including SessionCompletenessQuality for whether the user’s top-level goals were met and UserFrustrationMetric, run as online rules on production threads. Rules only evaluate data logged after you create them.

To measure task completion itself, see how to measure AI agent task success without inflating the number and the best tools to measure AI agent task completion. For the experience side, see resolution without satisfaction.

To write your first success or experience rule, see the violations guide.

Frequently asked questions

Why do users complain when my AI agent completes the task?

Because completing a task is not the same as helping. The agent may have solved an easier version of the request, taken so many steps that the user waited too long, or produced output the user had to check and rewrite. Each of those counts as a success and still feels like a failure to the user.

Is task completion rate a good metric for AI agents?

It is a good baseline and a poor finish line. It tells you whether the agent reaches an end state, which matters for a new agent. It does not tell you whether the user's actual goal was met, how efficiently, or whether they trust the agent more or less over time, so track it alongside friction and outcome signals.

How do I tell whether my agent's successes are real?

Read a sample of conversations marked successful and look for rephrasing before the success, corrections afterward, requests for a human, and users re-running the same task. Then check what your success flag actually measures. A returned response or an HTTP 200 is not a verified outcome.

How do I measure user satisfaction with an AI agent without surveys?

Use behavior in the conversation: rephrasing loops, polite exits right after a weak answer, follow-ups asking the agent to double-check its own work, and users delegating simpler tasks over time. These cover the users who never click a rating, which is most of them.