Why Your RAG Pipeline Is Failing (And It's Not the Retrieval)
Every team that ships a retrieval-augmented generation system eventually hits the same wall: the demo worked, the pilot looked promising, and then real users started asking real questions and the answers got worse. The instinct is almost always to blame retrieval — swap the embedding model, tune the chunk size, add a reranker. Sometimes that helps. Usually it doesn't, because the actual problem was never retrieval quality. It was that nobody defined, in a testable way, what a correct answer looks like.
The chunking trap
Chunk size gets blamed first because it's the easiest thing to tune and the easiest to convince yourself is the problem. In practice, most production RAG failures we've diagnosed have nothing to do with chunk boundaries. They come from a mismatch between how the source documents are structured and how the retrieval system treats them — a 40-page policy document split into 512-token chunks loses the section headers that gave each chunk its context. The fix isn't a smaller or bigger chunk size. It's chunking that preserves the structure a human would use to answer the same question.
What actually breaks in production
In our experience building RAG systems for clients, the failures cluster into a small number of categories, and none of them are "the vector database is bad":
- Queries that need information from multiple documents, when retrieval only pulls the top-k from one.
- Questions with a time dimension ("what changed since Q2") that pure semantic similarity can't express.
- Exact-match needs — a product SKU, a legal clause number — that dense embeddings blur together.
- No mechanism to say "I don't know" when nothing relevant was actually retrieved.
Every one of these is solvable, but not by tuning a similarity threshold. Multi-document synthesis needs a retrieval strategy that explicitly fans out across sources. Exact-match needs hybrid search — BM25 or a keyword index running alongside the vector search, not instead of it. And the "I don't know" problem needs a confidence gate before generation, not a hope that the model will hedge appropriately on its own.
The missing piece: an evaluation set
The teams that fix RAG quality fastest are the ones who stop debugging by vibes and build a small, real evaluation set first — twenty to fifty actual questions users have asked, each with a known-correct answer and the source passages that justify it. That set turns "the answers feel worse" into a number you can move. It also tends to reveal, immediately, that the problem was never retrieval at all — it was that the prompt let the model answer confidently from passages that didn't actually support the claim.
None of this is exotic. It's the same discipline that makes any software system reliable: define correct behavior, measure against it, and change one variable at a time. RAG doesn't get a pass on that just because there's a language model in the loop.