RAG Retrieval Held Up on 1,801 Papers, Then Changed at 2.68M Passages

BM25 barely slipped at 1,801 papers. A 2.68M-passage Wikipedia test changed the picture for RAG retrieval.

RAG Retrieval Held Up on 1,801 Papers, Then Changed at 2.68M Passages

On a collection of 100 research papers, BM25, a standard text search method, put a paper that answered the question in its top five results for 99.7 percent of questions. On 1,801 papers, that number was 98 percent. I built the test to find the threshold at which that number drops, and it barely moved.

That result looked like good news for anyone running retrieval augmented generation, or RAG, where an AI model answers a question using passages that a search step fetched first. If search stays good as the collection grows, the rest of the system has solid ground to stand on. Then I ran the same kind of test on 2.68 million Wikipedia passages, and the picture changed.

This article reports both tests as one story. The first test used questions from my earlier test of agentic retrieval. The second used public questions that nobody wrote for my setup.

The code, questions, labels, and results for every test are in a public repository.

What did I want to find out?

I wanted to know how often the search step of a RAG system, the part that finds documents, misses the right one as the collection grows. I also wanted to know whether it matters if the extra documents look like the right answer.

A distractor is a document that does not answer the question. Some distractors are random, and some are close in topic to the right document.

Each test asked the same plain question. If I add more distractors and change nothing else, does the search still find a document that answers the question?

Before running anything, I fixed a rule for what would count as a meaningful drop, so that I could not choose the bar after seeing the results. A size counted as a degradation point only if the score fell at least five points below the smallest collection. Five points means one more question in every 20 loses its answer, and a point here means one percentage point.

The rule also needed the drop to be clearly real and not luck. My tests use only some questions, so a different set of questions would give a slightly different drop. I checked this with the 95 percent range, which is the span where the true drop probably sits.

Think of a bathroom scale that can be wrong by about 2 kilograms either way. If it says you lost 1 kilogram, you cannot tell whether you lost anything. If it says you lost 8 kilograms, you really did.

My measured drop is the scale reading, and the 95 percent range is how jumpy the scale is. If that span includes zero, which means no change, the drop could be luck.

For example, a 6 point drop with a span from a 9 point drop to a 1 point gain does not count, because that span includes no change. A 6 point drop with a span from a 9 point drop to a 3 point drop counts, because every value in that span is a drop.

Three scores appear in this article: recall at 5, success at 10 and exact match. The first two ask whether search put a good passage near the top, and exact match asks whether the model’s answer matched an accepted answer. I define each one where it first matters.

Test one, 2,000 arXiv papers

The first collection held 2,000 recent arXiv abstracts in machine learning, language, and search. I used 298 questions, each written from one paper and checked by hand, so every question had a known right paper. No paper was used twice, so the 298 questions came from 298 different papers.

For each question I built collections of 100, 250, 500, 1,000 and 1,801 papers, and each larger collection contained the smaller one. The right paper was always inside. The rest were distractors of two kinds.

Every other paper in the 2,000 could be used as a distractor. That included the 1,702 papers with no question and the 297 papers that belonged to other questions. For any one question, those papers were documents that should not answer it, and I checked that next.

Ordinary distractors were picked at random from papers that were not close to the right one. I made three separate random picks, which I call draws.

Look-alike distractors were the papers nearest to the right one, taken in order of similarity. I measured similarity with a separate small embedding model, a tool that turns text into lists of numbers. That way the collections were not built to fool the search methods I was testing.

I tried two search methods, BM25 and embeddings. Embeddings rank passages by how close their lists of numbers are to the question’s list.

I ran each method in two ways, called flat chunks and parent documents, which makes four search setups. Flat chunks searched small windows of two sentences and returned the top five windows. Parent documents searched the same windows but returned the whole abstract for the top five distinct papers.

The four setups are BM25 with flat chunks, BM25 with parent documents, embeddings with flat chunks and embeddings with parent documents.

The score was recall at 5, which is the share of questions where a paper that answers the question appears in the top five results. A paper counted as an answer only if it passed a check I describe next.

How I checked that distractors did not answer the question

A look-alike paper can accidentally answer the question, which would make the search look worse than it is. So I used two AI models to screen the 50 closest papers for each question.

Claude Haiku 5.5 flagged papers that might answer the question, and Claude Sonnet 5.5 reviewed each flag. Haiku flagged 446 cases and Sonnet cleared 313 of them, but I removed every flagged paper to be safe.

Each removed paper was replaced by the next closest one, so collection sizes stayed exact. Removals used up part of the 2,000 papers, which is why the largest collection is 1,801 papers and not 2,000. One question was refused by one of the two AI models, so I checked its closest papers by hand.

Afterwards I read samples of the labels myself to check the models’ labels, including 40 papers the models never looked at, and all 40 were clearly off topic. I did not read every pair, so I treat these labels as a rough range. The calls to the two AI models cost 1.81 US dollars in total.

What happened

Table of recall at 5 for eight search setups on 298 arXiv questions, at 100 and at 1,801 papers. Recall at 5 fell between 1.0 and 3.4 points, with the largest drop for embeddings with flat chunks and ordinary distractors.

Table 1. Recall at 5 in percent on 298 questions, after screening. The change is between the smallest and the largest collection, and a negative number means the score fell. Look-alike distractors were run once, since they have no random draws, and ordinary distractors were averaged over three draws.

Dumbbell chart of recall at 5 at 100 papers and at 1,801 papers for four search setups and two kinds of distractor. Every setup lost between 1.0 and 3.4 points.

Figure 1. Recall at 5 at 100 papers and at 1,801 papers. The horizontal axis starts at 92 percent, so the gaps look larger than they are.

No setup met my rule. The biggest drop was 3.4 points, for embeddings with flat chunks and ordinary distractors. That is below my bar of five points.

But 3.4 is only what I measured on my 298 questions. Say you want to know how tall adults in a city are, and you measure 298 people. Their average is 170 centimeters.

You would not say the whole city averages exactly 170, because a different 298 people would give a slightly different number. You would say it is probably between 168 and 172.

My drop works the same way. The 3.4 is my 170, and the 95 percent range is my 168 to 172. For this setup the range ran from 1.3 to 5.4 points.

The top of that range is above five, so I cannot rule out a true drop just past my bar. I can only say that the drop I measured never reached it.

Parent documents ended 0.3 points higher than flat chunks in all four comparisons at 1,801 papers, meaning BM25 and embeddings with each kind of distractor. The ranges overlap heavily, so I would not call that a win. They also did not clearly reduce the damage from growth, since the drop was smaller in two comparisons, larger in one and the same in one.

The result held when I removed only the 133 papers that both models flagged, and when I dropped three questions whose paper did not actually support them. It also held when I dropped the eight questions that need only one fact from a paper, because no paper can partly answer them. In every version the largest drop stayed between 3.1 and 3.4 points.

Taken together, the retrieval design choices I tested did not behave consistently as the collection grew. Those choices were the search method, BM25 or embeddings, and what the search returns, flat chunks or parent documents.

Embeddings lost more than BM25 with ordinary distractors, but not clearly with look-alike ones. Parent documents did not clearly beat flat chunks either. This is one test of two search methods, so treat it as a reason to test your own setup and not as a rule.

One detail matters. Look-alike collections started lower, at 96.0 to 98.3 for the smallest size, probably because even 99 look-alike papers already compete hard with the right one. Growing the collection cost only one or two more points.

Why did that test feel too easy?

The test left very little room to fall. The scores began near 100, the collection stopped at 2,000 papers, and the biggest collection I could build for both kinds of distractor was 1,801. A real document store can be far larger.

The questions were also written from the paper they ask about, so they may share words with it. A search method has an easy job when the question and the answer use the same vocabulary. That could hide how it behaves on questions that people really type.

I did not want to leave the result at “no breaking point found”. So I looked for public questions that nobody had written from the answer, and a much larger collection.

Test two, 2.68 million Wikipedia passages

Natural Questions contains real, anonymized search queries, with answers marked on Wikipedia pages. BEIR packages 3,452 of those questions with 2,681,468 Wikipedia passages and human relevance judgments. A relevance judgment is a label saying a passage is known to help answer a question.

A passage that carries that label is what I call a judged passage. It works like the answer key at the back of a textbook. People marked these passages before I ran any search so that I could check the search against the key.

Passages without a label are not necessarily wrong. Nobody marked them either way, so an unlabeled passage can still contain an answer.

I set aside 40 questions for checking my code and used the other 3,412 for the main test. This larger, public test is where the picture from the first experiment changed.