How to track success rate for an AI agent
- Define success per user goal, not per message. “Resolved a billing question” and “booked a meeting” need different definitions. A reply was produced is never the definition.
- Pick how each outcome gets judged. Behavioral signals (no rephrasing, no human request, a follow-up action in the product), an LLM judge reading the whole conversation, explicit feedback, or a verified system state such as a confirmed booking. Most teams combine two.
- Run the judgment on production conversations. Test sets miss the messy, ambiguous requests real users send. Use online evaluators or rules that score live traffic.
- Report the rate per intent. “71% overall” hides the one workflow sitting far below the rest. The gap between your best and worst intent is your roadmap.
- Calibrate against conversations a person has read. Sample conversations each week, label them by hand, and check that the automated judgment agrees. Use explicit thumbs-down feedback as a second check.
- Compare before and after every change on the same intent and the same definition, so prompt and model changes are judged by outcomes rather than preference.
For a full walkthrough of writing success definitions, see how to measure AI agent task success without inflating the number.
Tools that do this
- Amplitude Agent Analytics. A built-in Task Completed signal on every agent session, next to User Friction and User Intent, feeding funnels and retention. The best fit if success rate needs to sit beside conversion in the same charts. Free up to 5,000 agent sessions a month.
- PostHog. A Task outcome LLM-judge template that labels conversations resolved, partly resolved, unresolved, or handed off, run per generation, trace, or session with sampling. Bring your own model key; each evaluation counts as an AI event, and the first 100,000 a month are free.
- Opik. An open-source (Apache-2.0) SessionCompletenessQuality judge that checks whether the user’s top-level goals were met, run as an online rule on production threads. Free to self-host, Pro cloud $19/month. Rules only score data logged after you create them.
- Datadog LLM Observability. A Goal Completeness evaluation template that checks whether the agent resolved the user’s intent across the full session. The natural choice if you already run Datadog APM. Free up to 40,000 LLM spans a month.
- Trodo. Task completion, satisfaction, and failure rate per job, where jobs are discovered by grouping traces by what people asked the agent to do, with Slack alerts when a job’s failure rate crosses a threshold. Pro is $249/month.
- Agnost AI. No built-in conversation success score. You define success as a plain-language rule (“complete the user’s request, or say clearly that it can’t be done”) and Agnost AI reports how many analyzed conversations broke it, with evidence attached; you can also mark interactions with the
successfield on events. Tool-call analytics report success rate, errors, and latency per tool, and Auto Improve measures outcomes after a fix ships. Free up to 1,000 events a month.
Splunk Agent Observability (formerly Galileo) has an Action Completion metric, Maxim a Task Success evaluator, and Confident AI Task Completion and Goal Accuracy metrics. For a side-by-side comparison, see the best tools to measure AI agent task completion.
Response is not resolution
Intent resolution rate is the share of conversations in which the user’s stated or implied goal was actually met. Every model produces a response. A fluent, well-formatted, confident answer can address the words the user typed and still miss the problem they had.
The measurement is hard because intent is rarely explicit. Users don’t fill in a form saying what they wanted and whether they got it. They rephrase, abandon, come back five minutes later and try again, or copy the output and use it. You infer resolution from that behavior or from a judge reading the conversation.
Three ways to judge resolution
Behavioral proxies (start here). No new instrumentation, only the conversations you already log. Failure signals:
- Rephrasing. The user’s next message restates the previous one in different words.
- Immediate re-query. The user starts a new conversation on the same topic within a few minutes.
- Drop-off without action. The conversation ends abruptly with no closing message and no follow-up action in the product.
- Escalation. The user asks for or is routed to a human.
Resolution signals are the reverse: a follow-up that builds on the answer rather than restating the question, a closing confirmation, or a product action showing the task got done. Proxies are noisy but directionally useful, and they give you a number to move.
LLM-as-judge (the scale option). Once volume outgrows reading, run each finished conversation through a separate model with the transcript, a description of your product, and a rubric. Anchor the rubric to the user’s goal: “Given that the user was trying to do X, was X done?” works far better than “Was this a good response?” Check the judge against human-labeled samples before you trust its trend. More in how to calibrate an LLM judge for agent monitoring.
Explicit signals (the calibration layer). A thumbs up or down, a “did this solve it?” prompt, or a checkpoint at the moment a task completes. Few users answer, and the ones who do skew toward strong reactions, so don’t make explicit feedback your main metric. Use it to check whether your proxies and judge would have flagged the same conversations.
Resolution means something different per product
| Product | What “resolved” usually means |
|---|---|
| Support agent | The user’s question was answered or their issue fixed without a human or a repeat contact |
| Coding assistant | The suggestion was accepted and kept |
| AI tutor | The learner showed understanding and moved on, instead of asking the same foundational question again |
| Companion | No discrete task: use whether the conversation led the user to return, and treat it as a weaker signal |
Define resolution in terms of the job your product is meant to do, then measure that.
Mistakes that make the number useless
- Substituting CSAT. Survey responses come from a small, skewed slice of users. The users who had a mediocre experience and quietly left are invisible.
- Using test-set accuracy. Production inputs are messier and more varied than any test set, so lab scores routinely overstate production resolution.
- Averaging across all conversations. “Resolution is 71%” says little. “84% for how-to questions, 43% for multi-step setup” tells you where to work.
- Counting the agent’s own claim of success. An agent saying “done” is not evidence the task happened. See how to detect when an AI agent claims success without taking action.
Once the rate is segmented by intent, use it three ways: prioritize the lowest-resolution intents, judge every prompt or model change by whether it lifted resolution on the intents where the old version was weakest, and compare user segments to find where the agent already works well. And a high resolution rate is not the whole story either; see why success rate can be high while users are unhappy.
To set up a success rule in Agnost AI, see the violations guide; to mark failed interactions in code, see the success field in the Python SDK guide.
Frequently asked questions
How do I measure intent resolution rate for my AI agent?
Classify each conversation by what the user wanted, define what resolved looks like for each of those goals, and judge every production conversation against that definition with behavioral signals, an LLM judge, or both. Report the rate per intent, not as one average, and check the judge against conversations a person has read.
Can Langfuse or LangSmith track agent success rate?
Yes, but you build the metric. Langfuse stores success as a score you define, filled in by an LLM judge, reviewers, or user feedback. LangSmith's multi-turn online evaluators can judge intent, outcome, and trajectory on production threads, but there is no prebuilt task-completion evaluator.
Does Agnost AI track agent success rate?
Not as a built-in conversation score. You define success with a plain-language rule checked against every analyzed conversation or with the success field on events, and Agnost AI reports tool-call success rate per tool out of the box. Auto Improve also measures outcomes after a fix is deployed.
Is LLM-as-judge reliable for measuring agent success?
It is reliable enough to track trends if you anchor the judge to your product's definition of success and check it regularly against conversations a person has labeled. A generic prompt like was this a good response drifts toward rewarding fluent answers. Ask whether the specific user goal was met instead.
