RAG is not dead, your retrieval is just bad

Every few months someone announces that retrieval-augmented generation is obsolete, usually on the grounds that context windows are now large enough to hold everything. Then we get called into an enterprise deployment where answer accuracy sits at 61%, the team has upgraded the model twice, and the problem is that the pricing table was chunked in half so no retrieved passage contains both the product and its price.

Long context did not make retrieval unnecessary. It made bad retrieval more expensive, because now you can stuff 200 pages into the prompt, pay for all of it, and still get the wrong answer — with worse latency.

Across the rescue engagements we have run on underperforming assistants, accuracy improvements have come from five retrieval changes. Not one of them required a better model.

One: chunk on structure, not on token count

The default in most tutorials is to split documents every 500 tokens with a 50-token overlap. It is easy, it is deterministic, and it destroys meaning at exactly the places that matter.

A fixed-size split severs tables from their headers, so a retrieved fragment contains numbers with no indication of what they measure. It separates a clause from the condition that governs it, so the assistant confidently states a policy that only applies in circumstances the chunk no longer mentions. It cuts procedures mid-sequence, producing steps 4 through 9 of something.

Chunk on document structure instead: sections and subsections as natural boundaries, whole tables kept intact with their headers repeated into each row-group chunk, list groups kept together, and procedures kept whole even when long. The test to apply is simple — read the chunk in isolation and ask whether it still means what it meant in the document. If it does not, the boundary is wrong.

Adding a small amount of context to each chunk helps more than its cost suggests: the document title, the section path, and the effective date, prepended as a short header. It makes a passage retrieved out of context interpretable, and it gives the embedding something to anchor on beyond the raw text.

Two: hybrid search, because vectors are bad at identifiers

Semantic search is excellent at the thing it was designed for: finding conceptually related passages when the wording differs. It is notably weak at exact tokens — product codes, error codes, policy numbers, version strings, part numbers. Embeddings place "ERR_4021" and "ERR_4012" in nearly the same place, which is precisely wrong.

This matters because real enterprise questions are full of identifiers. Customers paste error codes. Employees ask about a specific policy number. Support agents search by SKU. A pure vector system answers these questions with something plausible about a different product.

Running keyword search (BM25 or similar) alongside vector search and fusing the results catches both the paraphrased question and the identifier lookup. Reciprocal rank fusion is a reasonable default for combining them and requires no tuning. This change alone typically resolves an entire category of complaints that gets logged as "it does not understand technical questions".

Three: rerank, which is usually the biggest single win

First-stage retrieval optimises for recall — get the right passage somewhere in the top fifty. It is not good at ordering, because embedding similarity is a coarse proxy for relevance to a specific question.

A cross-encoder reranker takes the query and each candidate passage together and scores actual relevance. It is far more accurate than embedding similarity because it can attend to the interaction between question and passage rather than comparing two independently computed vectors. It is also slower, which is why it runs on the top 30–50 candidates rather than the whole corpus.

In our engagements this is consistently the single largest accuracy improvement available, and it takes about a day to implement using a hosted reranker. Teams skip it because it feels like an optimisation rather than a core component. It is a core component.

One practical note: rerank a larger candidate set than you intend to use, then take the top three to five after reranking. Retrieving five and reranking five accomplishes nothing — the reranker needs candidates to discriminate between.

Four: metadata, filtering and freshness

The most damaging retrieval failure is not returning nothing. It is returning something related but wrong: the policy for a different region, the previous version of the pricing sheet, the internal draft that was superseded eight months ago. The assistant then answers faithfully from the wrong source, with a citation, which makes it more convincing rather than less.

Every chunk should carry metadata: source document, effective date, region, product line, audience, and status. Queries filter on them. A customer in Singapore asking about returns should not be able to retrieve the EU returns policy at all — not merely rank it lower.

Freshness needs explicit handling rather than hoping recency correlates with relevance. Superseded documents should be excluded from the index rather than deprioritised within it, because "outranked" still means "retrievable". Where historical versions must remain accessible for audit, they belong in a separate index that ordinary queries do not touch.

This is also where a content audit earns its place. In one deployment over 4,000 support articles we found 340 documents that were contradictory or superseded. Removing them improved accuracy more than any single technical change, and no amount of retrieval engineering would have compensated for leaving them in.

Five: rewrite the query before you retrieve

Real questions are compressed, misspelled, and frequently depend on the previous turn. "What about for enterprise?" is a perfectly clear follow-up to a human and retrieves nothing useful as a standalone query.

A small, fast model rewriting the query with conversation context before retrieval resolves this cheaply. "What about for enterprise?" becomes "What is the data retention policy for enterprise plan customers?" — which retrieves correctly. The rewrite costs perhaps 80 milliseconds and a fraction of a paisa, and it fixes a large share of the complaints that present as "it forgets what we were talking about".

Generating two or three query variants and merging their results helps further on ambiguous questions, at proportionally more cost. We usually implement single rewriting first and add variants only where evaluation shows a specific class of question still failing.

The content problem no retrieval technique can fix

Everything above assumes the right answer exists somewhere in your corpus. Frequently it does not, or several contradictory versions do, and no amount of retrieval engineering resolves that.

In one deployment over 4,000 support articles, an audit found 340 documents that were either superseded or in direct conflict with another article. Removing them improved measured accuracy more than any single technical change we made. The system had been faithfully retrieving and answering from documents that a human expert would have known to ignore.

The audit is worth running before the build, and it looks for four things: contradictions between documents on the same topic, content with no effective date where the currency cannot be established, drafts and internal working documents that were never meant to be authoritative, and topics with no coverage at all. The last category is the most valuable output — it tells you what your assistant will not be able to answer before you launch it and discover the same thing through customer complaints.

This also changes who owns the assistant’s quality after launch. If accuracy is bounded by content, then the content owner rather than the engineering team holds the lever. We wire the weekly unanswered-question clustering directly to whoever writes the documentation, because that is the loop that improves the system.

When retrieval is the wrong architecture

Retrieval is the right default for question answering over a corpus. It is the wrong tool in three situations worth recognising early.

When the answer requires aggregation across the whole corpus rather than a passage from it. "How many of our policies mention indemnity caps?" is a query, not a retrieval. The right architecture extracts structured data from the documents once and answers from a database — retrieval will return five relevant documents and the model will guess at the count.

When the source of truth is a system rather than a document. Order status, account balance, availability, entitlements — these live in APIs and change constantly. Retrieving a document about how orders work is not an answer to where this order is. Assistants that need both should have both, with the model choosing the tool.

And when the corpus is small enough to fit in context comfortably and changes rarely. A hundred-page policy handbook can simply be supplied with a cached prompt. Building retrieval infrastructure for it adds latency, cost and a failure mode in exchange for nothing.

Measure retrieval separately from generation

The mistake underneath all of these is measuring only the final answer. When accuracy is 61%, an end-to-end score tells you nothing about whether the system retrieved the wrong passage or retrieved the right one and wrote a poor answer from it. Those have entirely different fixes.

Build the evaluation set from real questions with known correct answers — several hundred is plenty — and run it automatically on every change to chunking, embedding model, retrieval configuration, prompt or model. Without it, every improvement is a guess and every regression is discovered by a user.

Correct-refusal rate deserves particular attention because it is the metric that moves first when something has gone subtly wrong. An assistant whose refusal rate is falling while its answer rate rises has usually not got better — it has started filling gaps.

What this is worth

On the deployments where we have applied all five changes to an existing underperforming system, answer accuracy on real user questions has moved from the low 60s to the low-to-mid 90s. The model was unchanged in every case. In two of them the client had already upgraded models twice, at meaningful cost, with no measurable improvement — because the model was never the constraint.

RAG is not dead. It is a retrieval system with a language model attached, and most teams have spent their attention on the language model.