Prompt debates are too easy.
Someone thinks the agent should be warmer. Someone else wants it shorter. The founder wants it to sound “more premium.” Legal wants seven warnings. Support wants it to stop saying “I understand.”
Four people edit the prompt. Nobody checks production.
Amazing process. Very artisanal.
Prompting from production conversations means using real user-agent sessions to identify specific failure patterns, write targeted prompt changes, test them against real examples, and monitor whether user behavior improves after release.
It is less glamorous than clever prompt wording.
It works better.
^ prompt review meeting when nobody brought actual user conversations
Why opinion-led prompting fails
Opinions are not useless. Taste matters. Product judgment matters.
But prompts are behavior controls. Changing them because someone has a vibe is how you get prompt sediment: layers of advice, constraints, tone preferences, exceptions, and warnings that nobody can explain anymore.
Opinion-led prompt changes usually sound like:
- “Make it more concise.”
- “Be more helpful.”
- “Ask better questions.”
- “Do not hallucinate.”
- “Sound less robotic.”
Cool. Also, what failed?
Without production examples, the prompt change is a wish.
What production conversations give you
They give you evidence.
| Conversation pattern | Prompt insight |
|---|---|
| Users rephrase the same ask | Intent examples are missing |
| Agent answers first clause only | Need multi-intent handling |
| User says “not that” | Disambiguation failed |
| Agent escalates late | Risk threshold unclear |
| Agent repeats docs | Recovery behavior weak |
| User abandons after generic answer | Need context-specific next step |
Now the prompt change can be precise.
Bad task:
“Improve billing prompt.”
Better task:
“When a user asks about charge timing, require account lookup before linking cancellation docs.”
One is vibes. One is an actual fix.
^ yes this gif again, because apparently prompt docs become conspiracy boards naturally
The production prompt loop
Use this loop:
- Find repeated conversation failures.
- Pick one pattern.
- Write the smallest prompt change.
- Replay it against real failed sessions.
- Add the best examples to evals.
- Ship.
- Measure whether the production pattern moved.
Step 7 is where teams usually disappear into the fog.
If you cannot tell whether the pattern improved, you are not prompt engineering. You are prompt decorating.
Sorry.
The prompt diff should cite the conversation
One small habit helps a lot: every meaningful prompt change should link back to the production conversations that motivated it.
That does two things.
First, it keeps prompt edits grounded. The team can see the user language that exposed the gap. Second, it gives you a replay set for testing. When someone changes the prompt later, you can check whether the old failure comes back.
Prompt changes should look like this:
| Field | Example |
|---|---|
| Pattern | Users ask to “reverse charge” but agent treats it as policy question |
| Evidence | 17 billing conversations, 6 abandoned |
| Change | Add refund slang examples and require billing lookup |
| Expected movement | More correct refund tool calls, fewer docs-only replies |
That is a prompt change with a memory.
TLDR
Stop editing prompts from opinions alone.
Use production conversations to find repeated failures, write small targeted changes, test against real examples, and monitor user behavior after shipping.
Agnost helps teams close that loop by connecting production conversation patterns to the fixes they should actually make next.
FAQ
Should founders still review prompts?
Yes. Founder taste is useful. But it should be grounded in production examples.
How big should prompt changes be?
Small enough that you can tell what changed. Giant prompt rewrites make learning harder.
Should every production failure become an eval?
No. Repeated, costly, or high-risk failures should become evals. Weird one-offs can stay as notes.