Evaluating Retrieval Quality: Measuring What Reaches the Model

The answer is wrong, so someone rewrites the prompt. Still wrong. Someone swaps the embedding model. Still wrong. Someone changes the chunk size, adds a reranker, and rewrites the prompt again — three changes, one commit, no measurement, and no idea which change helped or whether any of them did.

I have spent roughly two decades building and debugging software systems, including search and retrieval pipelines, and the pattern I keep seeing is not that the prompt is the problem. It is that teams evaluate the pipeline as one blob. When you score only the final answer, retrieval quality and generation quality are fused into a single number, and a single number cannot tell you which stage to fix.

Retrieval is a separate system with its own inputs, outputs, and failure modes. It can be measured before the model ever sees a token. This article assumes you already know how to trace a RAG answer through ingestion, retrieval, assembly, and generation, and that you have identified retrieval as the suspect stage. Here we design the measurement: a query set, a relevance definition, a metric choice tied to your task, and a slicing habit that turns "the RAG is bad" into a specific, fixable defect.

Why Retrieval Needs Its Own Scoreboard

An end-to-end answer score is a composite. It mixes query understanding, retrieval, context assembly, and generation into one number. When that number drops, you have learned that something is wrong somewhere in a four-stage pipeline. That is not a diagnosis; it is a symptom with a bill attached.

Worse, the composite actively hides retrieval failure. A capable model can produce a fluent, well-grounded answer from the wrong evidence — it finds a plausible passage and reasons from it. It can also produce a correct answer from partially wrong evidence, because the right fact happened to survive in a chunk that ranked seventh. Both cases look like success on the end-to-end score and both are retrieval failures waiting to surface on the next query.

Retrieval evaluation is upstream and cheap. You can score retrieved chunks without calling a generation model at all, which means you can iterate in seconds instead of minutes, and you can run hundreds of configurations without paying for tokens. That cost asymmetry is the whole argument: measure the stage you can afford to measure often.

Name the decision your evaluation is meant to support before you build it. Are you deciding whether to keep or replace the embedding model? Whether to add a reranker? Whether to change chunking? Whether the query rewrite step is helping or drifting? Each decision implies a different metric and a different slice. An evaluation without a decision attached is a ritual.

Define What Counts as Relevant Before You Measure Anything

Relevance is a judgment, not a property of a document. Every metric downstream inherits its meaning from this choice, so make the choice explicit and write the rule down.

The first fork is binary versus graded. Binary labels — relevant or not — are enough when you only care whether the evidence is somewhere in the top-k. Graded labels matter when ranking order changes the outcome, because a system that puts the best evidence first and one that buries it at position nine are not equivalent even if both retrieved it.

The second fork is granularity. Document-level labels and chunk-level labels answer different questions, and a chunking change can invalidate a document-level label set entirely. If you label at the document level and then split documents into 400-token chunks, you no longer know which chunk carries the evidence.

Labels are the expensive part. You have three practical options, each with a different error profile:

• Hand-labeling a small set. Slow, but it is the closest thing to ground truth you will get, and it forces you to articulate the relevance rule.
• Deriving labels from known-good source documents. Fast, and it works when you already know which documents should answer which queries. It inherits whatever bias shaped that known-good set.
• LLM-as-judge relevance scoring. A practical shortcut when you have no labels at all. Treat it as a proxy, not a replacement: it correlates with human judgment rather than reproducing it, and it inherits the judge model's blind spots — including a tendency to reward surface-level topical overlap over actual answerability.

Watch the label sanity problem. If your ground truth says a document is relevant but no human would agree, every metric you compute afterward is confidently wrong. Some evaluation tooling exposes this directly as a "holes" or label-sanity check: documents marked relevant that never appear in any retrieved set, which usually means the label is stale or the index never contained the document in the first place. That check is worth running before you trust a single score.

Build a Query Set That Represents Real Traffic

The query set is the benchmark. Build it from the wrong distribution and you will optimize for a system nobody uses.

Sample from real user queries, support tickets, logs, or the actual task distribution — not from the documents you happen to have indexed. A query set derived from your corpus measures whether the corpus can find itself, which is a much easier question than whether it can answer your users.

Cover the query types your system actually faces. Four categories do most of the work:

• Single-fact lookup. One document holds the answer.
• Multi-hop questions. The answer requires combining evidence from several documents.
• Ambiguous or underspecified queries. The system must either ask, guess, or retrieve broadly.
• No-answer and out-of-scope queries. The corpus does not contain the answer at all.

Include the last category deliberately. A retrieval system that always returns something looks excellent on recall and fails in production, because it hands the model confident-looking evidence for questions the corpus cannot answer. If your evaluation set contains no unanswerable queries, it cannot detect that failure.

Stratify by difficulty and by domain so a single easy segment does not dominate the average. Then hold back a slice you do not tune against. If you iterate on the same queries repeatedly, you are fitting the benchmark, not improving the system — the same overfitting dynamic that shows up when embedding models are evaluated against public datasets they may have seen during training.

Size the set for the decision. A few dozen well-labeled queries can expose obvious failures and separate two embedding models on clear-cut cases. A few hundred is where segment-level conclusions become stable enough to trust. But these are starting heuristics, not statistical guarantees: confidence depends on effect size, query diversity, label variance, and how many slices you are cutting. Thirty queries cannot support broad conclusions across many segments. Start smaller than feels comfortable; a labeled set of thirty real queries beats an unlabeled set of three thousand.

Recall, Precision, and Ranking Metrics: Pick by Task, Not by Habit

Metrics come in two families, and the choice between them is a question about your downstream task, not about which one sounds more rigorous.

Rank-agnostic metrics ignore order. Recall@k answers the first question you should ask: did the evidence make it into the candidate set at all? This matters because a reranker can only reorder what the first stage retrieved. If recall is broken, no reranker will save you.

Rank-aware metrics reward putting the most relevant items first. NDCG@k is the standard example: it gives more credit for relevant results near the top and discounts them as they fall down the list. MRR@k is a narrower lens — it scores only the position of the first relevant result, which is the right metric when the task needs one correct answer fast, such as a lookup or an agent tool call, rather than a full ranked list.

Precision@k and F1@k answer a different question: how much of what you retrieved is actually useful. Precision matters when context budget is tight or when noise degrades generation — a model reasoning over nine irrelevant chunks and one relevant one is doing harder work than it needs to.

Here is the decision rule I use:

• If your context window comfortably holds the top-k results, optimize recall. The model can use anything in the window, so ranking within the window is secondary. This is the common case for retrieval pipelines feeding a reasonably sized context.
• If the window is tight, or the model attends unevenly across a long context, ranking quality starts to matter. Now NDCG@k earns its place.
• If the task needs the first correct result fast, use MRR@k. A full ranking is wasted information.

One useful signal: recall and NDCG often move together on the same data. A model with good recall on your corpus will usually have good NDCG too. So when the two diverge sharply, that divergence is itself worth investigating — it usually means your relevant items are being retrieved but ranked low. But do not treat that as a diagnosis of the scoring function by itself. Divergence can also come from label errors, candidate depth, query type, or metric construction. The divergence tells you where to look; the inspection tells you what is wrong.

Do not report a single number. Report the metric, the k, the label definition, and the query set. A recall@5 of 0.82 means nothing without the other three, and it is not comparable to anyone else's 0.82.

Measure Context Utility, Not Just Retrieval Correctness

Most retrieval evaluations stop at the candidate list. That is one step too early.

A retrieved chunk can be relevant and still useless. It may be redundant with three other chunks that say the same thing. It may be truncated mid-sentence because the chunk boundary fell in the wrong place. It may have lost the table header that gave its numbers meaning. In each case, the retrieval metric says "correct" and the model receives garbage.

Context utility asks a different question: does the assembled context contain the information needed to answer, and does it contain it without drowning the signal in near-duplicates? Retrieval correctness is a property of the candidate list. Context utility is a property of what you actually hand to the model. Keep the distinction visible, because they fail independently.

Three practical proxies:

• Coverage. For multi-hop queries, check whether all required facts are present in the assembled context, not just whether each source document was retrieved.
• Redundancy. Measure how much of the context is repeated content. Near-duplicate chunks consume budget and can push the model toward over-weighting a single claim.
• Judge-based relevance on the assembled context. Score the whole context against the query rather than scoring chunks individually. This catches assembly problems that per-chunk scoring misses.

This is where chunking decisions show up. A chunking change can improve recall while making context utility worse, by splitting evidence across boundaries so that no single chunk is self-contained. If you only measure recall, that change looks like a win.

Context utility is a downstream gate, not a retrieval metric. It sits after recall and ranking, and it detects failures that belong to chunking, deduplication, ordering, and assembly. The branch rule is simple: if candidate metrics are good but assembled context is poor, stop tuning retrieval and inspect assembly. If candidate metrics are poor, no amount of assembly work will fix it.

Slice the Results Until the Failure Has a Name

An average score of 0.8 can hide a segment at 0.3. The average is a summary; the slice is the diagnosis.

Cut the results along the dimensions that predict failure: query type, query length, document type or source, recency of the document, language, and whether the query needed one document or several. Then look specifically at the multi-hop and no-answer slices. These are where retrieval systems fail most often and where aggregate metrics are most forgiving, because they are usually a minority of traffic.

Then read the actual retrieved chunks for the worst slice. Read them. The metric tells you where to look; the text tells you why. This is the step people skip, and it is the step that produces the fix.

Common named failures worth separating:

• The evidence was never indexed — an ingestion problem wearing a retrieval costume.
• The embedding missed a paraphrase — the query and the document mean the same thing but do not look alike.
• The chunk boundary cut the answer in half.
• The reranker demoted the right document.
• The query rewrite drifted from the user's intent.

Each of these has a different fix. Without the slice and the read, they all look like "retrieval is bad," and you will fix the wrong one.

Keep a small failure gallery of real examples. It is the fastest artifact for explaining the problem to someone who did not run the evaluation, and it survives every model swap.

Offline Labels Versus Online Signals

Offline evaluation with labeled queries is interpretable and repeatable. Its reliability depends entirely on how representative the query set and labels are — which is why the earlier sections carry most of the weight.

Online evaluation observes real users interacting with the deployed system. It captures real information needs, which is something no offline set can fully simulate. But it only sees responses to the system you actually shipped, so comparing a new variant requires reasoning about what users would have done under it — a counterfactual problem, not a measurement problem.

Online signals are also noisier and slower to read. Clicks and dwell time reflect presentation, position bias, and user patience as much as retrieval quality. A result ranked first gets clicked more regardless of whether it deserved to.

The practical default for systems with enough traffic: use offline metrics to choose between candidate retrieval configurations, and use online signals to corroborate that the choice survived contact with real traffic. That corroboration only counts under an explicit experiment design — online behavior can reflect UI changes, policy shifts, or seasonality rather than retrieval quality. And for low-traffic or high-risk systems, a well-built offline set may be the only credible evidence you have. That is a legitimate reason to invest more in labeling, not a reason to skip evaluation.

Change One Thing at a Time

Freeze the query set and labels, then change exactly one component per run: embedding model, chunk size and overlap, reranker, query rewrite, or top-k.

Record the metric, the k, the configuration, and the date for every run. A retrieval experiment log is a reusable asset; a memory of what you tried is not. Six months from now, when someone proposes swapping the embedding model again, the log answers the question in thirty seconds.

Expect interactions. Chunk size and embedding model are not independent — a model trained on longer passages behaves differently at 200 tokens than at 800. And a reranker can only reorder what the first stage retrieved, so a reranker cannot fix a recall problem. If recall@k is low, adding a reranker is buying latency to reshuffle a list that does not contain the answer.

One-change-at-a-time is the debugging default because it gives clean attribution. But after you have isolated the obvious effects, test a small predeclared set of interactions or full pipeline configurations. You will not get the same clean attribution, but you will catch combinations that serial tuning misses.

Set a decision threshold before you run the experiment. How much improvement in which metric justifies the added latency, cost, or operational complexity of the change? Deciding after you see the numbers is how teams end up with a pipeline that is slower, more expensive, and no better.

Treat a regression on one slice as a real result, not noise, until you have enough queries to argue otherwise. And when the retrieval metrics are already good and the answers are still wrong, stop tuning retrieval. The bottleneck has moved to context assembly or generation, and continuing to tune retrieval is wasted effort.

A Minimum Viable Retrieval Evaluation

Here is the smallest version I would build, and it fits in an afternoon.

1. Collect 30 to 50 real queries, stratified by type, including multi-hop and no-answer cases.
2. Write binary relevance labels on the chunks or documents that should be retrieved, and write down the rule you used.
3. Compute recall@k first. It is the metric that tells you whether the evidence is reachable at all.
4. Add a rank-aware metric — NDCG@k or MRR@k — only if context budget or ordering matters for your task.
5. Add one context-utility check on the assembled context, even if it is a coarse judge-based score. Use it as a gate: if candidates look good but assembled context does not, inspect chunking and assembly before touching retrieval.
6. Slice by query type and read the retrieved text for the worst slice.
7. Log every configuration change with its metric, so the evaluation becomes a record rather than a ritual.

One concrete next move: have a second person apply your relevance guideline to a sample of queries where you disagreed with yourself or with the judge. Record the adjudication outcomes. That single exercise will tell you more about your label quality than any additional metric.

One boundary worth stating plainly. Public embedding benchmarks measure zero-shot performance on datasets you did not choose, and reported leaderboard scores can diverge from performance on your own unseen data — a gap that widens as models are repeatedly evaluated against the same public sets. Treat those benchmarks as a shortlist filter, not a decision. Your labeled query set is the decision.

Before you touch the embedding model, the reranker, the chunk size, or the prompt, build the smallest evaluation that can tell you which of them is actually wrong. Measure recall first. Slice until the failure has a name. Change one component per run. Keep the log.

A frozen query set with labels is a reusable asset. It keeps paying out every time the pipeline changes, every time a model is swapped, every time someone proposes a new chunking strategy. A prompt tweak is a one-off. The evaluation is the thing that compounds.