Every few weeks a team swaps an embedding model, bumps top_k, or bolts on a cross-encoder reranker, and the demo looks better. Nobody recorded a baseline, so nobody can say whether production retrieval actually improved — or quietly broke five query classes that used to work. That is the state of most RAG stacks in 2026: retrieval tuned by vibes, judged by anecdote.
The fix is unglamorous infrastructure work. Build a small, version-controlled evaluation set, score retrieval separately from generation, and wire the scores into CI so that a chunking change or model swap has to prove itself. Every other lever — chunk size, hybrid search, reranking, query expansion — only matters once you can measure it.
Split retrieval from generation — they fail for different reasons
A wrong answer has exactly two causes: the retriever never fetched the right evidence, or the generator ignored or contradicted what it got. If you only score the final answer you will spend a week tuning prompts against a retrieval bug. Oracle’s production RAG evaluation guidance makes the split explicit: judge retrieval and generated answers separately, and treat SQL / natural-language-to-SQL as its own route for current structured facts, tested for query correctness, permissions, and freshness rather than forced into document-retrieval metrics (Oracle Developers).
Retrieval metrics are the cheap half and they need no LLM: recall@k (did the required passage make the top k at all), MRR (how high did the first relevant chunk rank), nDCG@k (graded relevance, position-discounted), and precision@k when the context window is the bottleneck. They are deterministic, fast, and model-independent, which is exactly what you want in a build gate. Generation-side metrics — faithfulness, answer relevance, context precision and recall — are noisier: use an LLM judge for continuous screening and humans for acceptance and for disagreements.
Build the golden set from real queries, not imagined ones
Thirty to one hundred hand-labelled cases will out-catch thousands assembled carelessly. Each case needs the query as a user actually typed it, the chunk or document IDs that must be retrieved, and, where practical, a reference answer. Pull them from support and search logs rather than generating them from your own chunks: questions written backwards from the answer make the right chunk trivially retrievable, so the suite inherits the retriever’s blind spots instead of exposing them.
Cover five shapes deliberately:
- Identifier lookups — part numbers, SKUs, codes, citations
- Paraphrases — wording that does not echo the source document
- Multi-hop — answers that require evidence from two or more passages
- Unanswerable — the corpus genuinely cannot answer, so abstention is the correct behaviour
- Permission-negative — questions answerable only from content the asking user must not see
Keep the file in git as JSON or CSV, stamp a date on each label so staleness is visible, and treat label edits and threshold edits as reviewable diffs.
Pick a retrieval path with evidence
This is where the harness pays for itself. Oracle’s companion benchmark compares keyword, vector, and RRF hybrid retrieval against the same ground-truth question set rather than crowning a universal winner, and recent research backs the instinct to measure before you standardise. A benchmark of ten retrieval strategies over 23,088 financial queries and 7,318 mixed text-and-table documents found that a two-stage pipeline — hybrid retrieval plus neural reranking — reached Recall@5 of 0.816 and MRR@3 of 0.605, beating every single-stage method by a wide margin, while plain BM25 outperformed state-of-the-art dense retrieval on those documents. Query-expansion tricks such as HyDE and multi-query gave limited benefit on precise numeric questions, while contextual retrieval gained consistently (arXiv:2604.01733).
Translation for an agency stack: hybrid search plus a cross-encoder reranker is the configuration to test first, but “semantic search wins” is a hypothesis, not a law — especially when the corpus is full of identifiers, tables, and policy codes. The reranker is also usually the single biggest jump in the ladder, which is another reason to measure it instead of assuming it.
Sweep chunking on your own data
Chunk size interacts with embedding model, document structure, and query mix, so tutorials hand you someone else’s knee on someone else’s curve. Sweep token sizes (128, 256, 512, 1024) and plot recall@k against size, then ablate overlap fractions until the recall curve flattens. Declare whether k counts chunks or deduplicated parent documents — identical document-level recall can hide very different context quality — and compare the evidence supplied inside a fixed token budget. If your vectors live in Postgres, the index-size and re-embedding cost side of that decision is the subject of our pgvector write-up, and where those vectors should live at all is covered in why most RAG stacks don’t need a managed vector database.
Wire it into CI so regressions can’t ship
An eval suite you run on demand is a report card. One that runs on every pull request touching chunking, prompts, embedding models, or retrieval parameters is a gate. Keep thresholds in a reviewed baseline file, fail the build when recall@k or faithfulness drops past tolerance, and never quietly lower the bar to get a merge through — a drop means revert or fix. Failures should be hard errors, not warnings nobody reads.
Add a slower scheduled pass over a sample of current production data: corpus growth and connector changes shift rankings in ways a frozen per-PR sample cannot see. Instrument the retrieval half too, because parsing and metadata filters quietly destroy recall before ranking ever starts — a failure mode we mapped in Why Your RAG Retrieval Fails.
Close the loop with production logs
Sample production queries on a schedule, cluster the failures by intent, and promote the instructive ones into the golden set with labels. That is the compounding part: every incident becomes a fixture, and an intent cluster with consistently poor retrieval is next quarter’s ingestion work — discovered by measurement instead of by a client escalation. Keep the layers straight while you do it: retrieval metrics tell you the evidence never arrived, and only the parser and filter metrics tell you why.
Start this week
- Export 30 real questions from logs and label the required chunk IDs by hand — budget roughly a day per hundred cases.
- Implement recall@k and MRR in plain Python. No vendor, no judge model, no API key.
- Pin the numbers to a
baseline.jsonand fail the build on a regression beyond tolerance. - Run exactly one experiment — hybrid versus dense, or add a reranker — and keep the change only if the golden set says it is better.
Vector databases were never the hard part of RAG; knowing whether retrieval works is. If you would rather not guess which half of your pipeline is broken, Outdoor Tek builds and audits retrieval infrastructure for marketing and support agents.