How a RAG Reranker Really Works: The Honest Explanation

When RAG retrieval disappoints, the advice AI engineers hear today is almost always “add a reranker.” Ask why a reranker works, and the answer usually stays at the architecture level: it is a cross-encoder, it applies attention over the query and the passage together, it is fine-tuned on relevance labels. All of that is true, and none of it says what the model actually learned. Push one level down, to terms a business partner could check, and the explanation usually stops.

That gap matters. A team that cannot say in plain terms what the reranker does cannot defend the choice to use one, and cannot spot the cases where a keyword lookup would beat it for a fraction of the cost.

This article gives the honest answer — the one you can hand to your business partner without waving hands. The reranker is not smarter than the embeddings step below it. It runs the same mechanism (statistical token association from training data), just conditioned differently (on the query-passage pair rather than each text independently). Once you see that, the “when to use a reranker” question stops being “add it because the tutorial did” and becomes “add it only when this specific tradeoff is worth paying for.”

🧭 New to the series? Start with the map: Prompt, Context, Loop sets out the three engineering layers every RAG system is built on — the prompt (the call itself), the context (what fills the model’s window), the loop (when the next call fires and when it stops) — and walks the whole series through that lens, article by article.

This article sits in Part I, alongside the embeddings triptych (2A / 2B / 2C). — Image by author

📓 Try the reranker on your own PDF at doc-intel/notebooks-vol1. The companion notebook loads a cross-encoder, applies it to a keyword-filtered top-K, and shows both the score and the tokens driving it. Change the query, watch which keywords carry the ranking.

1. What Data Scientists Say, and Why It Isn’t Enough

Ask three data scientists what a reranker does and you get three answers, roughly:

1. “It’s a cross-encoder. It scores the query-passage pair jointly and gives a relevance score.” Technically true, but the words cross-encoder and relevance are hiding what the model actually learned.
2. “It applies attention over both texts, so it sees the interaction between them.” True at the architecture level, but architecture does not tell you what the model is doing with that attention.
3. “It’s trained on relevance labels, so it learns which passages answer which questions.” Very close, but “learns which passages answer” is the wrong verb. The model does not learn to answer. It learns which tokens co-occurred.

None of the three is wrong. All three are incomplete in a way that matters when you have to decide whether to keep the reranker in your pipeline, whether to fine-tune it on your corpus, or whether to replace it with something cheaper.

The rest of this article walks that answer down to the mechanism, then names three consequences that change how you architect enterprise RAG.

2. What Actually Happens Inside a Reranker

The reranker is a specific kind of transformer, trained on a specific kind of data, that produces a specific kind of number. Each of those three pieces matters.

2.1 The Architecture: Cross-Encoder, Not Bi-Encoder

An embedder (bi-encoder) reads the query alone, produces one vector. Reads a passage alone, produces one vector. Compares the two vectors by cosine. Each text is embedded independently, and the model never sees them together during scoring.

A reranker (cross-encoder) reads the query and the passage together, as one concatenated input: [CLS] query [SEP] passage [SEP]. It runs BERT-style attention over the joint input, where every token can attend to every other token. It outputs a single relevance score.

That “reads them together” is the whole architectural difference. Bi-encoder: two vectors, one comparison operation. Cross-encoder: one forward pass, one score. The joint attention is why the reranker feels smarter, and why it is 30 to 100 times slower per query.

2.2 The Training Data: MS MARCO and Its Cousins

Where does the reranker learn its scoring? From query-passage relevance pairs labeled by humans. The canonical dataset is MS MARCO (Bajaj et al. 2016, one million real Bing search queries with human-graded passage relevance). Others include Natural Questions (Google search + Wikipedia paragraphs), BEIR (a benchmark aggregator), and TREC.

Every training example is a triple: (query, passage, relevance_label). The model sees millions of these, and its weights adjust so that pairs labeled relevant get higher scores than pairs labeled not relevant.

That is the sole learning signal. The model is never shown a question and asked to compose an answer; it is shown pairs, and it optimizes for a score that separates relevant pairs from non-relevant ones.

Which raises the honest question: what pattern actually separates them in the training data?

2.3 What the Model Really Learns: Keyword Co-occurrence at the Pair Level

Here is the level down that rarely gets explained.

The model looks at millions of (query, passage, relevance) triples and asks: what patterns in the joint token stream predict the relevance label? The dominant pattern is not “answering.” It is which query tokens tend to co-occur with which passage tokens in high-relevance pairs.

Concretely, in MS MARCO the query “how to cancel my subscription” is labeled relevant against passages containing cancel, subscription, unsubscribe, terminate, end your membership. Millions of examples reinforce that when the query contains cancel, passages containing terminate or unsubscribe tend to be labeled relevant. The reranker’s weights absorb that association.

So the “smart” reranker is doing keyword linking at the query-passage pair level. It is a learned association table between query token neighborhoods and passage token neighborhoods, dressed up as a neural network score.

The embedder does the same thing, but at each text independently. The reranker does it conditioned on the pair. Same mechanism, different conditioning.

Second-order signals the reranker also picks up include positional patterns (a term appearing early in the passage often correlates with relevance), syntactic structure (subject-verb-object relations that link query tokens to passage tokens), and the presence of definitional phrasing (“X is Y”). Those help, but they are second-order; the dominant signal is keyword co-occurrence.

Why this frame matters: once you see the mechanism, the “will it work on my corpus?” question has a clear answer. If your corpus vocabulary and query vocabulary look like MS MARCO (general English, common web topics), the trained associations transfer, and the reranker feels magical. If your corpus vocabulary is specialized (insurance contracts, medical records, regulatory filings), the trained associations do not cover your domain, and the reranker inherits the same out-of-vocabulary failures as the embedder below it. No amount of “but it’s a cross-encoder” fixes that.

3. The Mechanism Shown: Where the Reranker Wins, Where It Hits a Wall

Section 2 made a claim: the reranker is a learned association table between question-language and answer-language. That claim is testable. Take a handful of candidates, score them with three embedders (MiniLM, ada-002, text-embedding-3-large) and three cross-encoders (bge-base, bge-large, ms-marco-MiniLM), and read each row.

3.1 Where It Wins: The Answer That Does Not Repeat the Question

Ask “What is the maximum coverage amount?” against three passages: the answer (“Cover is capped at 50,000 euros per year”), an echo that repeats the question’s words without answering (“The maximum coverage amount can be found in the benefits schedule”), and a distractor.

Every embedder ranks the echo first; both bge rerankers flip the answer to the top. — Image by author

Every embedder puts the echo first. It shares maximum, coverage, amount with the question, so its vector sits close. The answer shares almost nothing lexically, so it lands second or third. The two bge rerankers flip it: they read the question and the answer together, recognize that a “capped at X per year” passage answers a “maximum coverage amount” question, and lift it to #1. This is the reranker doing its one real job — bridging the question’s words to the answer’s words.

It is not a one-off. The same flip reproduces on plain factoids:

Same shape, general-knowledge version. bge lifts the answer over the echo; ms-marco keeps the echo on top. — Image by author

Across a dozen queries of this shape (who wrote a play, the boiling point of water, the speed of light, the first president, plus the enterprise trio of deductible, notice period, coverage) the two bge rerankers rescue the answer to #1 where every embedder ranked an echo above it. The win is real and repeatable, on exactly one shape: a short factual answer that does not repeat the question, sitting behind an echo that does.

Two honest caveats sit in the same two figures. First, not every reranker does it: ms-marco-MiniLM keeps the echo on top in both cases, the same lexical bias an embedder has. Second, when a strong embedder already answers the question (text-embedding-3-large gets several of these on its own), the reranker adds nothing over just using a better embedder.

3.2 Where It Hits a Wall: Your Private Vocabulary

Now the case that decides the enterprise question. Ask “what’s the rule on contractor overtime?” where the answer uses the company’s own term — “non-employee labor compensated beyond 40h/week” — and never the word contractor.

The answer never says “contractor,” so neither the embedders nor the rerankers can bridge the gap — the association simply does not exist in MS MARCO. — Image by author

No model surfaces the correct answer. The embedders miss it because the vectors for “contractor overtime” and “non-employee labor compensated beyond 40h/week” were never trained to sit close together in your domain. The rerankers miss it for the same reason: “contractor” and “non-employee labor” were never paired at high relevance in MS MARCO, so the cross-encoder has no association to fire. The joint attention runs, but it finds nothing to link.

This is the wall. When your corpus vocabulary diverges from general web English, the reranker’s learned associations do not transfer. Adding a cross-encoder on top of a bi-encoder does not fix an out-of-vocabulary problem — it inherits it. The fix is domain adaptation: fine-tuning both the embedder and the reranker on labeled pairs drawn from your actual corpus, so the model builds the associations your vocabulary requires.

4. Three Consequences for Enterprise RAG Architecture

Once you hold the mechanism clearly, three architectural decisions become straightforward.

First, the reranker is not a substitute for a good retrieval stage. It reorders a candidate set; it does not retrieve. If the correct passage is not in the top-K that the embedder returns, the reranker cannot rescue it. The embedder’s recall ceiling is the reranker’s ceiling too. Spend on retrieval quality before spending on reranking.

Second, the speedup math favors a hybrid first stage. The cross-encoder is 30–100× slower than cosine similarity. Running it over a full corpus is impractical. The standard pattern — BM25 or embedding retrieval to top-100, then reranker to top-5 — works because the reranker’s cost is paid only over a small candidate set. If your latency budget is tight, check whether a better embedder alone closes the gap before adding the reranker’s latency overhead.

Third, domain vocabulary is the deciding variable. On general-English corpora, a pretrained reranker adds measurable lift over embeddings alone, specifically on the echo-vs-answer shape described above. On specialized corpora with private terminology, a pretrained reranker adds little and may add nothing. The decision rule is simple: sample twenty queries from your actual users, build the echo-vs-answer test, and measure. If the reranker flips the ranking on most of them, keep it. If it does not, fine-tune or skip it.

The reranker is a useful tool with a narrow job: bridging question vocabulary to answer vocabulary when the two do not share surface tokens, in a domain the model has seen during training. Know that, and you can make the call on evidence rather than habit.