Kimi K3's 1M Token Context Window Tested Against RAG

With a context window of one million tokens, a natural question arises: can you skip RAG entirely?

The math looks tempting. Everything published over the last two years adds up to 127,068 tokens — twelve percent of the window. That makes it possible to dump all of it into the prompt instead of building chunks, computing embeddings, and worrying about retrieval quality.

But “it fits” and “it works better” are two different things.

That is exactly what this experiment measures. The same twelve questions are answered once through a RAG setup and once through the full corpus in the context window (referred to here as the long_context setup). The article also covers what went wrong along the way, which turned out to be almost as instructive as the results themselves.

This is the setup used:

• The same prompts: Both paths get exactly the same system instruction. The only thing that differs is what comes before it. For the RAG setup that is five selected text passages; for the long_context setup it is all 32 articles.
• The same model: Both paths run on Kimi K3 with the same generation settings. Temperature is the only setting that could not be chosen freely, so it is fixed at temperature = 1.
• A blind evaluation at the end: The two answers to each question were shuffled and presented as “X” and “Y” in an Excel file, with no hint which path produced which. The key sits in a separate file opened only after grading was complete.

That way the conditions were identical for RAG and for the full corpus, and the blind grading prevented self-influence.

→ 🤓 Find the full code in the GitHub Repo 🤓 ←

1 — Why the question looks different now

RAG (Retrieval Augmented Generation) came about as an answer to a technical limitation: a model could only see a few thousand tokens at a time. If you had more material than that, you had to pick out the relevant passages first and pass only those on. That works well, but it adds another component to the system that has to be maintained, tuned, and debugged.

This is where Kimi K3 comes in. The model by Moonshot AI offers a window of one million tokens. Other models with similarly large context windows also exist. This removes one of the main reasons why many people have used RAG. Other advantages still cited in favour of RAG include:

1. It’s cheaper.
2. It’s faster. Latency is lower.
3. It gives you traceable sources.

You can read more about this topic in the paper by Google DeepMind. Those three claims are what this experiment tests, on a corpus where every answer can be judged by the person who wrote it.

Hint for Newbies: A context window is the amount of text a model can read at once in a single request. Anything that does not fit has to be selected beforehand. That selection is exactly what RAG does.

2 — The setup: One corpus, two paths

The corpus consists of 32 files holding 33 articles from Medium and Towards Data Science, totalling 127,068 tokens. Token count was done with tiktoken and cl100k_base, not with Kimi’s own tokenizer. What Moonshot billed later was 127,346 input tokens per request, which also includes the system instruction and the question — close enough to plan with.

Three topics appear twice, once as the Medium version and once as the TDS version. At the time, Towards Data Science was still a publication on Medium rather than the separate platform it is today, which is how the same topic ended up in two versions. That duplication is intentional, since one of the questions targets it directly.

The fact that there were 33 articles rather than 32 was not noticed until grading: two articles had ended up in the same file, and both models pointed this out independently.

# One shared instruction for both paths. If the prompts differed, any quality
# gap could come from the wording instead of from the retrieval strategy.
SYSTEM_PROMPT = (
    "You answer questions about a collection of articles written by one author. "
    "Use only the provided article text. If the text does not contain the answer, "
    "say so plainly instead of guessing. When you state a fact, name the article "
    "title it comes from."
)

The RAG path splits the articles into 788 chunks of 900 characters with 150 characters of overlap, embeds them with all-MiniLM-L6-v2, and sends the five most similar chunks to the model together with the question. That comes to roughly 1,200 tokens per request.

The long_context path sends all 32 articles along with every single question — 127,346 tokens per request, roughly a hundred times as much.

One detail determines cost: the corpus always comes before the question in the prompt. Prefix caching only works as long as the beginning of the message stays identical character for character. If the question came first, every single call would be a cache miss and the whole experiment would be several times more expensive.

Hint for Newbies: You can picture prefix caching like a colleague you hand the same thick folder to every time, with a different question each time. The first time they have to read all of it; after that they still know what is in there. That is what the provider does in the background automatically. The important word is prefix: only the beginning is stored, and only as long as it stays identical character for character. If anything changes at the beginning, everything that follows is a cache miss. That is why the corpus comes first and the question last. In terms of cost, this results in a difference between $0.30 and $3.00 per million input tokens.

# The context always comes first and the question last. That order matters:
# prefix caching only works while the beginning of the message stays identical
# across calls. Putting the question first would make every call a cache miss.
messages = [
    {"role": "system", "content": config.SYSTEM_PROMPT},
    {"role": "user", "content": f"{context}\n\n---\n\nQuestion: {question}"},
]

3 — 12 questions in three difficulty levels

Before the first call went out, twelve questions were defined, split into three groups of four. Those groups are the actual point of the experiment.

Group A, single-fact: The answer sits in exactly one place in exactly one article — for example, which embedding model was used in the chunk size experiment and why. This is what retrieval is built for.

Group B, cross-article: The answer requires two or three sources at the same time — for example, when RAG is recommended versus fine-tuning, including places where the author contradicts themselves.

Group C, corpus-wide: The answer requires that the model has seen everything. One example is the question of how many articles link to a GitHub repo, and which ones.

For Group C the outcome is fairly predictable: five chunks cannot possibly answer a question that spans all 32 articles. The group still belongs in the experiment, because what is interesting is not whether RAG loses here, but how it loses. Does the model honestly say the information is missing, or does it guess and sound convincing while doing so?

4 — Evaluation: Why I graded blind

Instead of grading with a second model as judge, that approach was deliberately avoided — an LLM judge would only have added another source of error. With a corpus of one’s own texts, the author is the more accurate evaluator, because they know what is in there.

To prevent self-influence, a small script (make_grading_sheet.py) turns the run into a grading sheet: the two answers to each question are shuffled and written out as A1-X and A1-Y, with no indication which path produced them. The key ends up in a separate file that stays closed until grading is complete.

The assessment was not entirely blind, however. Some responses gave themselves away with phrases such as “based on the text chunks I have” — wording that can only come from a system working with text snippets. At those points it was clear a RAG response was being evaluated. In a future iteration, such self-revealing phrases would be stripped before assessment.

Grading used three criteria, with 0 to 2 points each:

The difference between the first and third criterion is the most important one in the whole setup. While correctness asks whether something is true, grounded asks where it comes from. An answer can be factually correct and still be drawn from the model’s general world knowledge rather than from the provided text — which matters when the goal is to evaluate retrieval faithfulness rather than general model capability.

5 — Three things that went wrong

Running the experiment surfaced several practical issues worth documenting. First, the tokenizer mismatch between tiktoken’s cl100k_base and Kimi’s internal tokenizer meant the pre-experiment token count was an estimate rather than an exact figure, though the 278-token difference turned out to be negligible for planning purposes.

Second, the blind grading was partially compromised by self-revealing phrases in the RAG responses. Removing such model-generated metadata before evaluation would make future comparisons cleaner.

Third, two articles had been placed in the same file without realising it, making the corpus 33 articles rather than the assumed 32. Both models caught this independently during grading — a useful reminder that LLMs can surface structural issues in a corpus that a human curator overlooks.

6 — Results

The long_context setup performed better on Group B and Group C questions, where cross-article synthesis or full-corpus coverage was required. RAG performed comparably on Group A single-fact questions, where the relevant passage was retrieved cleanly by the embedding search.

On cost and latency, the difference is significant. The RAG path uses approximately 1,200 tokens per request. The long_context path uses around 127,346 tokens per request — roughly 100 times more. With prefix caching active, the effective cost per cached token drops to $0.30 per million rather than $3.00 per million, which narrows the gap considerably but does not close it. Latency for the long_context path was measurably higher, consistent with processing a much larger input.

7 — When to use which

RAG remains the better default when cost and latency are constraints, when the corpus is larger than any available context window, or when traceable, chunk-level source attribution is a hard requirement. It is also easier to update incrementally — adding new documents means re-embedding only the new chunks, not re-sending the entire corpus on every request.

Long-context works better when the corpus fits comfortably within the window, when questions require synthesis across many sources, or when retrieval quality is hard to tune because the relevant information is spread diffusely. It is also simpler to operate: no chunking strategy, no embedding model, no vector store to maintain.

The two approaches are not mutually exclusive. A tiered system — RAG for the majority of queries, long-context for the subset that requires full-corpus reasoning — can combine the cost efficiency of retrieval with the coverage of a large window.

Final Thoughts

A one-million-token context window changes the RAG conversation but does not end it. For a corpus of 127,068 tokens, stuffing everything into the prompt is technically feasible and produces better answers on questions that require broad synthesis. For single-fact lookups, RAG is competitive and substantially cheaper. The right choice depends on the question type, the corpus size, the cost budget, and how much engineering overhead is acceptable. Running a small, blind experiment on your own data — rather than relying on benchmarks — remains the most reliable way to find out which approach fits your specific situation.