An agent’s completion claim should be checked against an authoritative result for the same action, object, and customer. A sentence such as “Your refund has been processed” is not evidence that a refund happened. Neither is an HTTP 200 from a tool that only queued the request.
The useful failure category is an unsupported completion claim: the assistant reports a finished action without sufficient evidence that the action reached the required state.
What evidence establishes completion?
For a refund, record the requested action, a stable order or refund identifier, the customer or tenant, and a receipt from the payment workflow. Define which state your product treats as complete. Accepted, submitted, and settled may mean different things.
| Agent answer | Execution evidence | Assessment |
|---|---|---|
| “Your refund is complete.” | Matching receipt in the required terminal state | Supported |
| “Your refund is complete.” | Matching request is pending | Premature completion claim |
| “Your refund is complete.” | Receipt belongs to another order | Wrong evidence |
| “I submitted your refund request.” | Matching request is pending | Potentially accurate; assess the exact promise |
| “Your refund is complete.” | Tool telemetry is missing | Unknown until the business system is checked |
That last distinction matters. Missing telemetry is not proof of a missing action. Your detector should separate an observed absence in a complete record from a record that might be incomplete.
A small executable check
The following example uses structured claims and receipts. These are an application-level contract, not fields a language model or tracing library automatically provides.
claim = {"action": "refund", "entity": "order_42", "tenant": "tenant_a"}
receipt = {
"action": "refund", "entity": "order_42", "tenant": "tenant_a",
"state": "pending"
}
def completion_supported(claim, receipt):
return bool(receipt) and all(
receipt.get(key) == claim[key]
for key in ("action", "entity", "tenant")
) and receipt.get("state") == "completed"
assert completion_supported(claim, receipt) is False
assert completion_supported(claim, {**receipt, "state": "completed"}) is True
In production, obtain the receipt from trusted application code or a verified provider response. Letting the model invent both the claim and its receipt defeats the check. If you extract claims from free text with a model, evaluate that extraction separately. “I can refund you” is not a statement that a refund already occurred.
Download our synthetic fixtures and Python detector for examples of missing actions, pending operations, and receipts attached to the wrong order. They run locally without credentials or network access.
Where to put the check
A monitoring rule can find unsupported claims after a run. A response gate can prevent some claims from reaching the user. These are different responsibilities.
For important actions, build the final status response from application state. The agent can explain the result, but it should not decide whether the payment succeeded. If the operation remains pending, expose a status reference and explain what happens next. If the system cannot determine the outcome, say so and reconcile it before retrying.
The MCP tool specification distinguishes tool execution errors from protocol errors. That is another reason a successful transport response alone cannot establish a successful action.
Measure the detector before alerting
Label a review set with three outcomes: supported completion, unsupported completion, and insufficient evidence. Include negations, future promises, partial completion, and asynchronous operations. Check the detector’s precision among findings and its misses among labeled failures.
Count eligible completion claims in the denominator. Dividing failures by every chat message makes the rate look smaller when unrelated traffic grows. Track evidence coverage alongside the rate so a broken instrumentation path cannot masquerade as an improvement.
Instrument and alert on unsupported claims
Unsupported completion is one kind of silent wrong answer. For detectors that cover the others, such as answers that contradict retrieval, stale sources, wrong-tenant entities, and user corrections, see how to catch wrong answers that don’t show up as errors in traces.
Useful application-level attributes
OpenTelemetry captures execution structure, but product outcomes often require custom attributes. Use names your organization owns rather than presenting them as standard semantic conventions.
app.agent.release = "2026.10.28.2"
app.agent.intent = "refund_order"
app.agent.outcome = "pending"
app.agent.receipt_id = "refund_r42"
app.agent.evidence_complete = true
app.agent.user_confirmed = false
app.agent.manual_edits_count = 0
Do not attach secrets or unrestricted personal information. If a receipt identifier is sensitive, hash or tokenize it consistently so equality checks still work without exposing the raw value.
Alert on rates with evidence coverage
For an unsupported-completion alert, calculate:
unsupported_rate = unsupported_claims / eligible_completion_claims
evidence_coverage = claims_with_complete_receipt_data / eligible_completion_claims
Alerting on the first metric without the second is dangerous. If receipt collection breaks, unsupported findings may fall because the detector abstains—or rise because missing evidence is incorrectly treated as failure. Graph both and route incomplete evidence to an instrumentation queue.
For the broader workflow, continue with the production AI agent monitoring guide.
Investigate the cause in Agnost
Use tool-call records to inspect what executed and conversation evidence to inspect what was promised. Supply the application outcome if it is needed; Agnost cannot infer an unseen payment ledger reliably from prose.
The fix may be a response-state check, a missing tool permission, or an asynchronous status handler. A stronger prompt is only appropriate when the underlying execution contract already works. For retries that could repeat a real action, continue with idempotency for agent tools.
