Agnost AIAgnost AIBook a demo
Open +
← All articles

How do I measure the quality of AI-to-human handoffs?

Measure handoff quality per escalated conversation on five checks: timing, context passed, routing, what the user was told, and whether the human had to re-ask anything. Score each with a written rule or LLM judge and report the failure rate per check. Agnost AI evaluates such rules on conversations; Dimension Labs and PostHog offer escalation signals.

How to measure handoff quality

  1. Confirm the handoff happened first. Quality only applies to transfers that reached a human. Requested, attempted, accepted, and responded are separate states; how to verify an AI agent actually handed off to a human covers that lifecycle. Score quality on accepted handoffs, and count missing ones separately.
  2. Keep both halves of the conversation together. You can’t judge context or momentum without the human’s first replies. Send the AI turns and the human follow-up under one conversation ID, or store the helpdesk ticket ID as metadata so the two can be joined.
  3. Score five checks per handoff. Timing, context, routing, user clarity, and momentum (table below). Each should be a yes or no question that can be answered from the transcript.
  4. Write each check as a rule. For example: “Do not ask the user for information they already provided.” “Before transferring, tell the user what happens next.” “Escalate when the user reports a safety issue.” In Agnost AI these are violations, evaluated against each analyzed conversation with the supporting messages attached.
  5. Report a failure rate per check, not one blended score. “Context missing in 22 of 140 accepted handoffs” tells you what to fix. A single 0 to 100 score hides which dimension moved. If you need one number, use the share of handoffs that pass all five checks.
  6. Count the handoffs that should have happened. A conversation where the user asked for a person twice and never got one is a handoff failure too. Track missed escalations next to bad ones.
  7. Read the evidence, fix, and compare. Open a sample of failing handoffs, change the escalation prompt or the payload, then compare each check’s failure rate over matched windows.

The five checks

Check Question to answer from the transcript Typical failure
Timing Did the agent escalate before trust broke? Escalated after three loops and an “are you even reading this?”
Context Did the human get the actual issue and history? Summary says “user needs help with billing”
Routing Did it reach the right team for the current request? Routed on the first intent, not the one the user ended on
User clarity Did the user know what happens next and roughly when? “Connecting you now” followed by silence
Momentum Did the next step happen without repeated work? Human asks for the workspace ID the user already gave

A handoff can look successful in backend events while the transcript shows otherwise:

User: “I already said the workspace ID above.”

Agent: “I am connecting you with support. Please provide your workspace ID.”

That preserved the ticket, not the experience. It passes a “did the agent escalate” check and fails the momentum check.

Timing cuts both ways

Late handoff is the obvious failure: the agent loops, apologizes, asks for the same information again, and escalates after the user has lost patience. Early handoff fails too. If the agent escalates every ambiguous request, users learn it is a slow support form, human queues fill, and automation never earns trust.

Handoff timing What the user thinks How to detect it
Too early “Why did I use the agent at all?” Escalation on an intent the agent has tools to handle, with no failed attempt first
Too late “Why did it waste my time?” Repeated questions, corrections, or human requests before the escalation
Right “This knows when to get help” One failed path, then escalation with context

For deciding the escalation triggers themselves, see when should an AI agent escalate to a human.

The quiet handoff failures

The most damaging handoff failures don’t produce angry tickets:

  • The summary is generic or describes the wrong issue.
  • The summary misses that the user is frustrated or the issue is urgent.
  • The human asks for information already provided.
  • Routing uses the user’s first intent, not their current one.
  • The user gets no expectation of when a person will reply.
  • The handoff triggers after a loop instead of before it.

Each one teaches the user to skip the agent next time. That shows up later as falling usage, not as a handoff metric, which is why it pays to check the transcript instead of the escalation flag. The same behavioral signals that reveal unreported friction apply here; see what tool shows where users get stuck with my AI agent.

Tools that do this

  • Agnost AI. Evaluates handoff rules written in plain language (“escalate when the user reports a safety issue”, “don’t re-ask for information the user gave”) against every analyzed conversation, links each violation to the messages behind it, and alerts when a rule crosses a threshold. Routing tools show up in tool-call analytics with success rate and latency, and handoffs are recorded as events when your framework emits them. There is no built-in handoff-quality score; the checks above are rules you define.
  • Dimension Labs (dashbot.io now redirects there). A Human-Escalation Attempt dimension plus containment and escalation analytics, with failure modes ranked by escalations and repeat contacts. The better fit for contact-center teams on platforms such as Intercom, Ada, Cognigy, or Genesys. From $299/month, no free tier.
  • PostHog. A “Task outcome” LLM-judge template that labels sessions resolved, partly resolved, unresolved, or handed off, so you can count handoffs and write custom judges for their quality. The better fit if your product analytics already runs on PostHog.
  • Gleap. Its Kai Support report shows the share of conversations handed to the team, alongside answer rate and ratings. It only covers Gleap’s own Kai agent, not an agent you built, and AI agent reports start on the Pro plan ($299/month billed yearly).

To start, add your first handoff rule under Violations after setting up the quickstart.

Frequently asked questions

How do I know if my AI agent hands off to a human at the right time?

Look at what happened before each escalation. If the user had already repeated themselves, corrected the agent, or asked for a person more than once, the handoff was late. If the agent escalated a request it had the tools to handle, it was early. Track both rates per intent.

Is a high escalation rate bad?

Not by itself. A high rate can mean the agent is risk-aware and escalates the right cases. The questions that matter are whether escalations happen for the right reasons, at the right moment, and with enough context that the human doesn't start over.

What should an AI-to-human handoff summary include?

The user's goal, the steps already tried and their results, the relevant account or object references, and the open question the human needs to answer. Check summaries against the conversation, because a confident wrong summary can leave the human worse off than no summary.

What is the fastest way to improve handoff quality?

Find conversations where the human asks for information the user already gave the agent. That repeated-question failure is easy to detect, easy to fix in the handoff payload, and the one users notice most.