When I was brought in to investigate the accuracy failures of a client’s AI assistant, everyone told me the exact same story
It worked perfectly during the executive demo. It even seemed fine for the first two weeks in production. Then it started showing cracks.
Upon investigation here is how they demoed it
The engineering team had brought leadership into the room, typed “How do refunds usually work on our platform?”, and watched a vector-backed Retrieval-Augmented Generation (RAG) system return a flawless summary within two seconds. It cited policy docs, listed processing windows, and earned immediate rollout approval.
Two weeks after launching to active merchant users, support tickets were piling up and executive trust was collapsing.
A merchant had logged into their portal and entered a real operational query
Why was transaction TXN-48291 declined, and has the refund for it posted yet?
The system returned a confident, step-by-step breakdown of why the transaction failed and stated that a $140.00 credit was posted yesterday morning. It looked authoritative. It read cleanly.
There was only one problem
TXN-48291 was never refunded.
When I audited the retrieval pipeline, the root cause became obvious. The system did not degrade two weeks after launch. It was set to fail from day one. It simply took fourteen days for real users to start asking questions that exposed the underlying structural gap.
Prototype/Demo Queries (Easy ~20%)
“How do refunds usually work on our platform?”
“Summarize our terms for chargeback disputes.”
Fuzzy, conceptual language → Vector Similarity Search Works Perfectly (~95% Success)
Production Queries (Hard ~80%)
“Why was transaction TXN-48291 declined, and has the refund posted?”
“Which fee schedule applies to merchant M-902 under international clearing?”
Exact transaction IDs & live ledger integers → Vector Search Fails (Wrong status returned)
Semantic Similarity vs Structural Exactness
Most AI prototypes pass internal reviews because demo queries naturally cluster around what vector search does well, fuzzy linguistic closeness.
Vector search embeds text chunks into high-dimensional mathematical space. When a user asks a conceptual question like “How do refunds work?” the embedding engine easily finds chunks containing “merchant refund processing terms” or “standard return windows.” The vocabulary matches, the cosine similarity score is high, and the LLM synthesizes a clean response.
Production environments do not ask conceptual questions. They ask for operational truth.
When a query demands structural exactness, vector search breaks down across two physical failure modes
Semantic similarity scores vocabulary closeness, not transaction compatibility
An embedding model has no concept of non-interchangeable financial records or primary keys. To an embedding model, the text describing a declined card error on one transaction and a settled refund on another looks almost identical. The vector engine returns a high similarity score for the wrong transaction because the surrounding sentences share the exact same vocabulary.
Live account state is a dynamic integer, not a static concept
Ledger balances and refund statuses sit inside an SQL database or core banking system, changing second by second. They do not exist inside a static PDF or Markdown file stored in a vector index. Vector search either hallucinates a status based on historical text or admits it cannot query the database.
The demo sampled the easy region of the query distribution. Production sampled all of it.
Here are 3 audits you can run today
If your AI feature is stalling between a successful staging demo and production adoption, run these three diagnostic checks against your system today.
1. Retrieval Accuracy
Symptoms - High success on general queries, dangerous hallucinations on exact IDs.
Root Cause - Using vector similarity for questions requiring structural graph traversal.
Do a Query Split test
Pull 10 real support tickets, user logs, or operational inquiries from the last month. Do not use queries picked by the engineering team for a demo.
Split those 10 queries into two piles
Pile A (Conceptual/Fuzzy) - Broad questions like “How does our chargeback policy work?” or “Summarize the onboarding steps for new merchant accounts.”
Pile B (Exact/Structural) - Questions containing specific transaction IDs, account numbers, error codes, or requests for live data (“Has the settlement for TXN-48291 cleared yet?”).
Run both piles through your RAG system.
If Pile A achieves a high accuracy score while Pile B produces wrong transaction details, missing statuses, or hallucinated numbers, your failure mode is structural. You are attempting to solve exact relational lookups using vocabulary proximity.
2. Scaled Latency
Symptoms - Sub-100ms in staging, multi-second spikes under production document volume.
Root Cause - Unindexed scans or pointer-chasing over unoptimized vector indexes.
Do a Load Testing against Realistic Volume
Measure retrieval latency against your actual production document count and concurrent user volume, not the 50 sample PDFs sitting in your staging environment.
Staging environments hide infrastructural bottlenecks. A flat vector search or an unindexed query responds in 30 milliseconds when scanning 1,000 document chunks.
When that same index scales to 500,000 chunks across millions of transaction logs and merchant accounts, memory bandwidth chokes and latency spikes into seconds. If you did not load-test retrieval performance at 10x your expected document volume, your production latency figures are an illusion.
3. Cost Per Query
Symptoms - Token bills exploding linearly with active user adoption.
Root Cause - Routing 100% of queries unconditionally to expensive frontier models.
Check Cost Per Query vs Cost Per Token
Examine your API dashboard and calculate what percentage of incoming user queries are routed directly to a top-tier frontier model ($15+ per million tokens).
In our analysis of token economics and agentic loops, we showed that enterprise teams using naive, unrouted architectures pay a massive “token tax.” Unconditional routing forces a $15/1M token model to handle trivial tasks like parsing a regex or reading a transaction ID.
Teams that implement tiered routing, directing routine lookups to local models, database queries, or cheaper Flash-tier endpoints, and escalating only complex reasoning to frontier models, reduce their blended API costs by a median 87.4%.
If every query in your pipeline hits the same frontier model regardless of complexity, your cost model will choke as soon as active user adoption spikes.
What it takes to fix each failure mode
Moving an AI feature from a fragile prototype to a production-grade system requires targeted architectural corrections
Retrieval Accuracy Fix
Flat Vector Store Only (Vocabulary proximity) → Knowledge Graph / Relational Grounding (Merchant → Transaction → Ledger → RefundStatus)
Move beyond flat vector stores for operational data. Map your domain entities into a relational or Knowledge Graph structure (Merchant -[:INITIATED]-> Transaction -[:HAS_LEDGER_ENTRY]-> LedgerStatus). When a query contains an exact identifier like TXN-48291, bypass vector embeddings entirely and execute a deterministic database or graph lookup.
Scaled Latency Fix
Brute-force Flat Scans (Thrashes RAM/Disk) → Entity-First Deterministic Traversals (Bypasses vector search when exact keys exist)
Implement Entity-First Deterministic Traversals. Anchor your queries directly onto primary keys in your database or graph index, reserving vector search strictly as an entry point for unstructured, natural-language queries without explicit identifiers.
Cost Control Fix
Single-Model Pipeline (All queries → Frontier) → Tiered Complexity Router (Edge / DB Lookup → Flash Model → Frontier Escalation)
Build a Complexity Router at the front door of your system. Classify incoming user requests before sending them to an LLM. Simple database lookups or exact status fetches should execute at near-zero API cost; only multi-step, ambiguous reasoning tasks should escalate to frontier models.
Conclusion
A working prototype proves that an LLM can understand your domain. It does not prove that your architecture can serve your users safely.
Prototypes evaluate best-case scenarios under controlled conditions. Production subjects your system to the full complexity of your operational environment, where wrong answers destroy user trust, high latency frustrates merchants, and unconstrained token usage drains engineering budgets.
Stop testing your AI features against the easy ~20% of queries that look great in an executive demo. Test them against the exact, structured, high-stakes questions your team faces every day, and build the underlying architecture required to answer them correctly.
This three-part diagnostic is the framework I use when evaluating production AI architectures for engineering organizations stuck between prototype and scale. If your team is struggling with hallucinated outputs on exact queries, high latency, or spiraling API invoices, feel free to reach out for a structured architectural audit.
Open questions can hide problems with exact or structured requests. I’d test real production cases before trusting a demo result. How did you find the gap?