The ticket didn't look urgent. A customer had followed our support assistant's instructions to set up an integration, it hadn't worked, and they were annoyed because the steps had seemed completely clear.
They were clear. They were also for the wrong version of our API.
The customer was on the current API, and the assistant had confidently walked them through setup for the deprecated one — endpoints that no longer existed, a payload shape we'd changed a year earlier — in the same calm tone it used for everything else. Every sentence was supported by our documentation. It was just supported by the wrong documentation.
That's what made the incident interesting. The model hadn't hallucinated; given the context we'd handed it, the answer was reasonable. The failure had happened earlier, in retrieval — and the uncomfortable part was that several of the decisions we'd made to keep retrieval fast and cheap at scale had made that failure easier to produce.
What we'd actually built
The system was a customer-facing documentation assistant over a multi-tenant SaaS product — RAG across public documentation, API references, and an internal knowledge base. At the time we were handling tens of thousands of queries a day, with a corpus in the low hundreds of thousands of chunks once every version and product area was included.
The pipeline was fairly standard and orchestrated in LangChain: ingest documents, chunk them, generate embeddings, store the vectors in Qdrant, then embed the user's question and retrieve the nearest chunks with cosine similarity before sending them to the model. Vanilla dense-retrieval RAG. It worked well at first.
The interesting part was what happened as the corpus and traffic grew. We had latency targets, memory limits, and an ingestion bill, so we started tuning. We pushed toward larger, uniform chunks so the index held fewer vectors, trimmed metadata we thought we didn't need, enabled scalar quantization in Qdrant to reduce vector memory, lowered HNSW search-time ef to improve latency, and removed the cross-encoder reranker from the hot path because it added both latency and per-query cost.
None of those decisions was absurd. That was the problem. Each one looked reasonable in isolation. The system stayed fast, memory stayed under control, the latency graphs looked good — and our hardest retrieval cases got progressively less forgiving.
It looked like a hallucination. It wasn't.
My first reaction was the obvious one: the model hallucinated. Tighten the prompt, add another instruction telling it to answer only from retrieved context, maybe move the request onto a stronger model. I probably would have wasted a week doing that if I hadn't opened the trace first.
In the trace, the model had been given a chunk containing the old API setup instructions, and it had summarized that chunk accurately. The model wasn't the component that had gone off-script. We'd handed it the wrong evidence.
That changed how I looked at the incident. When a RAG answer is wrong, generation is the failure you can see, so it's easy to blame the model. But if retrieval quietly selects the wrong source, the model often has no reason to know anything is wrong — it just turns bad evidence into good prose. So I stopped touching the prompt and went upstream.
The real problem started with chunking
The strongest root cause turned out to be embarrassingly simple: our documentation already contained the information we needed. A section might start with API v1 — deprecated, followed by several paragraphs of setup instructions. But our larger fixed-size chunks had sometimes separated those instructions from the heading that explained which version they belonged to, and we'd trimmed some metadata as part of keeping the index lean.
So by the time the chunks reached the vector store, the retriever might see "Configure the integration using endpoint X…" without ever seeing "This applies to deprecated API v1." A v1 setup chunk and a v2 setup chunk were now almost identical pieces of text. We had removed the one piece of context that actually mattered, and once we'd done that, everything downstream had a harder job.
The scale optimizations made a hard query harder
Quantization wasn't the root cause, but it made an already narrow distinction less forgiving. The v1 and v2 instructions were extremely close in embedding space — they differed on a few exact details that mattered enormously to the user but represented only a small difference semantically. Using int8 scalar quantization saved us a lot of memory, and on normal queries we saw little difference; but on near-duplicate documents, reduced vector precision could flip the ordering of candidates that were already almost tied.
The same was true of HNSW ef. We'd lowered search-time ef to keep p99 latency down, which means exploring fewer candidates during approximate nearest-neighbor search. For easy queries that trade-off was fine. For the awkward near-duplicate cases, it increased the chance that the right document wouldn't make it into the candidate set at all.
You can't rerank a result you never retrieved. Except we'd removed the reranker too. That decision had made sense on the latency spreadsheet, but a cross-encoder gave us another opportunity to compare the query directly against the candidate chunks before passing them to the model. Without it, whatever dense retrieval ranked highest was effectively what the model saw.
None of these choices alone explains the incident. The real failure was cumulative: we'd removed useful context during chunking, made approximate search more aggressive, reduced vector precision, and then removed the stage that could have recovered a poor first-pass ordering. The system still met every infrastructure target we'd set for it. We just hadn't set a target for retrieval correctness.
Hard constraints shouldn't be left to embeddings
The first fix wasn't a better embedding model. It was admitting that some things should never have been semantic decisions in the first place. Version is one of them. So is tenant. So are product, language, access level, entitlement, and sometimes region. If a customer is on API v2, I don't want vector similarity deciding whether v1 is "close enough" — I want v1 excluded.
So we moved to structure-aware chunking and started preserving the information the document itself considered important: headings, product area, API version, tenant scope, and lifecycle status. Those values became Qdrant payload fields, so at query time we could apply hard filters first and run semantic search only inside the set of documents that were actually eligible.
That single change eliminated most of the wrong-version behavior, and it left me with a principle I've kept since: if something is a business constraint, don't ask embeddings to infer it. Filter on it.
We stopped treating ingestion as append-only
The incident also exposed a second problem: our corpus had effectively become append-only. A new API version went in and the old version stayed. A page was replaced and its old chunks could still sit in the index unless something explicitly removed them. That's manageable when every document is clearly versioned, and dangerous when stale and current content are nearly identical.
So document lifecycle became part of retrieval correctness. Deprecated content was marked as such and filtered out of current-version queries, and when documentation was replaced, ingestion had to invalidate or update the corresponding chunks instead of simply adding another copy beside them. RAG quality isn't only about search. It's also about whether the corpus you're searching is in a sane state.
Rebuilding the retrieval stack
With the constraints and the corpus fixed, the rest of the work was rebuilding the retrieval path itself so it stopped losing on the hard cases — without putting all the cost we'd cut back onto every query.
Hybrid retrieval. Dense embeddings are good at meaning, but the things breaking us were often exact tokens: v2, endpoint names, error codes, configuration keys. Those distinctions can matter enormously while barely changing the semantic meaning of a sentence. So instead of relying on dense vectors alone, we combined dense retrieval with a sparse keyword signal and fused the rankings. The dense side still handled questions phrased differently from the docs; the sparse side finally gave literal version numbers, endpoints, and error codes the weight they deserved. That made a real difference on the cases where two documents meant almost the same thing but only one contained the exact identifier the customer had asked about.
Reranking, brought back selectively. We reintroduced a cross-encoder reranker, but not across the whole corpus because the first retrieval stage still needed to be cheap. Instead, hybrid retrieval produced a wider shortlist and the reranker scored only those candidates before we decided which chunks reached the model. Retrieval casts the wide net; reranking decides which few results deserve the expensive attention. That gave us back much of the ranking quality we'd lost without putting the full reranking cost onto every part of the pipeline.
Quantization, kept but measured. We didn't remove quantization either, since the memory savings were genuinely useful. Instead we enabled rescoring: use the quantized vectors to retrieve candidates quickly, then compare the top candidates against full-precision vectors before final ranking. We took the same approach with ef, raising it until retrieval recall on our difficult queries recovered and then looking for latency savings elsewhere, rather than blindly pushing ANN search faster.
That difference sounds small, but it changed how we made every decision afterward. Before the incident, we'd tune something, ask a few questions manually, and decide it looked fine. Afterward, every knob had a retrieval number attached to it.
Sometimes the right answer is "I don't know"
The last safeguard was permission not to answer. Previously, retrieval always returned something, and the model would usually turn that something into an answer — which is dangerous. Now, if nothing clears a relevance threshold after filtering, retrieval, and reranking, the system stops pretending the nearest available chunk must be correct. For a support assistant, "I don't have enough information to answer that reliably" is a better outcome than a confident answer built on weak evidence.
We also added citations to every answer. That helped users verify what they were being told, but it helped us just as much: a wrong answer with a source attached immediately tells you what the system believed, while a wrong answer without one leaves you guessing.
We couldn't fix what we couldn't measure
At some point during all this I had to admit I'd been tuning retrieval by feel — change the chunking, ask a few questions, read the answers, decide it seemed better. That's not a useful testing strategy once you're serving tens of thousands of queries, because the sample any engineer checks manually is tiny compared with the real traffic distribution.
So I built an eval set, and the important part was evaluating retrieval and generation separately. For retrieval, I paired real questions with the chunks or documents that should answer them and measured Recall@k and ranking quality after reranking, so that when I changed chunk size, quantization, hybrid-search weights, or ef, I could see whether the right evidence was still being retrieved before the model ever touched it.
For generation, we measured whether answers were actually supported by the retrieved context and whether they answered the question, leaning on existing RAG evaluation tooling for some of it. The exact framework wasn't the point; separating the stages was. A great generation score can't rescue bad retrieval, and perfect retrieval doesn't guarantee the model will use the evidence correctly. Those are different failures and they need different measurements.
I deliberately loaded the eval set with the cases most likely to break: v1 versus v2, two almost-identical documents, questions our docs didn't answer, ambiguous phrasing, exact endpoint names, and cases pulled from old support tickets. The happy path had never been our problem; the production bug lived in the tail.
That eval set became the most useful thing to come out of the incident. Before it, every retrieval change involved some amount of faith. After it, a change produced a number. If someone asked whether the reranker actually helped, we had a before-and-after. If a chunking change hurt Recall@k, we knew before shipping. If a model started answering questions the corpus couldn't support, we could see that too.
Eventually those evals became part of CI, so a documentation change, chunking change, embedding-model swap, or retrieval-tuning change ran through the hard set before release. It didn't make the system perfect. It gave us a way to know when we were making it worse, which is a much more useful property.
If you take one thing from this
When a RAG system gives me a confident wrong answer now, retrieval is the first place I look. Not because models never hallucinate — they do — but because a model can also give you a perfectly grounded, beautifully written, completely wrong answer if you retrieved the wrong source.
That's what happened to us, and the failure wasn't one broken library or one bad parameter. We had gradually traded retrieval margin for memory, latency, and cost without measuring what those trades were doing to the queries that were hardest to get right.
The fix wasn't to abandon optimization. It was to stop pretending performance knobs and correctness knobs were separate things. Chunking affects retrieval. Metadata affects retrieval. ANN configuration affects retrieval. Quantization can affect retrieval. Reranking affects retrieval. Corpus lifecycle affects retrieval. And deterministic business rules like tenant or API version should constrain retrieval before semantic ranking even starts.
You can build RAG that's fast and cheap at scale. But if you can't tell me your retrieval recall on a real set of difficult, near-duplicate queries, I wouldn't start by changing the model — I'd go find that number first.
Our chatbot didn't hallucinate the wrong API documentation. It read exactly what we gave it.
That was the problem.