How Kimi K3 Was Built: Inside the Technical Report

Frontier models are released with limited visibility into how they were built. The model and its system card are available, but the critical engineering decisions remain proprietary: the training process, the reinforcement learning objectives, and the optimisations that reduced serving cost. Consequently, most external analyses are necessarily based on the model’s observable behaviour rather than its construction.

Kimi K3 is the exception worth reading. It’s open, it sits close to the best closed models on most benchmarks, and Moonshot published a forty-seven page technical report that walks through the parts other labs keep private. I read it over an afternoon. What stayed with me is how small a share of the work is the model itself.

One caveat up front. Everything specific below is Moonshot’s. I assume the closed labs do their own versions of the same categories of work, but I’m inferring that from the outside, so read the wider “this is what the frontier looks like” claims as a guess.

The Architecture Is a Stack of Small Changes

The architecture is the obvious place to start, and it holds the fewest surprises. K3 is a 2.8-trillion-parameter mixture-of-experts model, and what lifts it over the last Kimi is three fairly ordinary engineering changes stacked together.

The attention keeps a fixed-size running state instead of a cache that grows with the input, which is what makes a million-token context affordable; most layers use that cheap version, and every fourth does full attention. Positions aren’t encoded explicitly, so the recurrence has to carry them, which lets the model stretch to a million tokens without the usual rescaling hacks. Each layer can also look back at every layer beneath it, not only the one directly below, so early signal doesn’t wash out on the way up. And each token is routed to 16 of 896 experts, sparser than before, which takes real care to keep stable.

Moonshot reports a 2.5× gain in scaling efficiency over Kimi K2, roughly the same quality for under half the training compute. It credits the architecture together with refined data and training recipes, without apportioning the gain between them, so don’t read the 2.5× as three architecture changes multiplying out. Each piece is an ordinary, well-tested idea rather than a single breakthrough.

The Three Changes in Plain English

Each of the three is easy to hold in your head once you drop the notation, and the plain version is where the intuition lives.

Attention is a lookup table, and that’s why long context is expensive. A transformer handles each token by looking back over every earlier token and pulling a weighted blend of them — a soft lookup. To do that it keeps a small key-and-value record for every token so far, the KV cache. Think of a filing cabinet that never throws a card away: each new word files a card, then flicks through every card already in the drawer to decide what to attend to. Fine at a few thousand words. At a million, the drawer is enormous and every new word riffles the entire thing, so the cost climbs with the square of the length.

K3’s main attention swaps the cabinet for a single running summary — a fixed-size notepad it writes over as it goes. The notepad never grows, and that fixed size is what makes a million-token context affordable. What you give up is that a summary can’t keep everything, so it has to forget, and K3 hands it a per-feature dial for how fast old detail fades. A summary also can’t recall an exact earlier token on demand, so K3 keeps one true cabinet layer for every three notepad layers and buys the precise recall back where it counts.

Figure 1: Two ways to mix over tokens. Full attention stores a key/value entry per token and compares each new token against all of them (cost ∝ n²); Kimi Delta Attention folds tokens into one fixed-size state through a forget gate α (cost ∝ n). K3 stacks three linear layers per global layer.

The fixed-state layers already track order as they update, which lets K3 drop explicit positional encodings entirely. Most models bolt position onto attention with something like RoPE, and stretching the context window later means rescaling those frequencies or interpolating them, which is fiddly and lossy. K3’s recurrence carries position for free, so the same weights run at 8K during early training and at 1M after the long-context stage with no positional surgery in between.

The same lookup trick, one level up. Stack a lot of layers and each one normally adds its output to a shared running total that flows up the network. Picture that total as a notepad passed up a line of people, each scribbling a line. By the top the first few lines are buried, and the network burns capacity just keeping early information alive. Attention Residuals let a layer reach back and read earlier layers’ outputs directly, weighting them with a softmax — the same soft-lookup machinery as attention but pointed across depth instead of across the sequence. A layer pulls a blend from source instead of playing telephone up the stack. The cost is memory rather than new machinery: every layer’s output has to stay live for the ones above it, so K3 attends over a handful of block summaries instead of all of its ninety-odd layers to keep that bill down.

Figure 2: Passing information up through the layers. A standard residual stream carries a single running total, so early-layer signal is buried by the top; Attention Residuals instead let each layer read earlier layers’ outputs directly, weighting them with a softmax (the α on the arrows): the same soft-lookup idea applied across depth instead of across tokens.

Mixture of experts routes each token to a handful of specialists. Rather than push every token through one giant feed-forward network, K3 keeps 896 smaller expert networks and a router that hands each token to just 16 of them. Picture a hospital with 896 specialists and a triage desk. No patient sees everyone; the desk picks the few who fit. That’s how the model can hold 2.8 trillion parameters and still only run 104 billion for any given token. Capacity and per-token cost come apart, so you can add experts to widen the model’s range without paying to run them all. The saving is in compute, not memory: all 2.8 trillion parameters still have to sit in fast memory to be reachable, which is part of why serving a model this size is a cluster problem. The snag is that routers play favourites. Leave it alone and it learns to funnel most tokens to a handful of star experts, which jam up while the rest sit idle and undertrained. So a working MoE has to force the load flat, the way a maître d’ spreads diners across all the waiters instead of seating every table in one section. K3 uses a rule that nudges each expert toward an equal share, and wrestling with that balance is a good part of why big sparse models are twitchy to train.

Figure 3: A big panel and a selective router. Each token is routed to 16 of 896 experts, so a 2.8T-parameter model runs only 104B per token. Left unmanaged, the router overloads a few experts and starves the rest; a balancing rule spreads the load evenly.

It comes down to three ideas: soft lookup, which turns up over tokens and again over layers; a fixed-size running state that stands in for that lookup wherever it got too expensive; and sparsity, which lets the model grow wide without growing its per-token cost.

Making a Serial Recurrence Run on Parallel Hardware

The running state that makes Kimi Delta Attention (KDA) cheap creates its own problem, and how Moonshot handles it is one of the more interesting parts of the report. A recurrence is serial by nature: the state after token t depends on the state after token t−1, so the naive implementation walks the sequence one token at a time. A GPU is the opposite kind of machine, built to run thousands of operations at once, and a strict left-to-right loop leaves nearly all of it idle.

The way out is a chunkwise formulation. Split the sequence into chunks of a few hundred tokens. Inside a chunk, the recurrence can be rewritten as a couple of dense matrix multiplications, which is exactly what the GPU’s tensor cores are fast at, so everything within a chunk runs in parallel. Only the small, fixed-size state has to pass from one chunk to the next, and that hand-off is the sole serial step left. Moonshot’s kernel then overlaps the hand-off with the next chunk’s matrix work, so the cores rarely stall waiting on it. Almost all of the cost turns into parallel matmul, and the serial part shrinks to a sliver of the runtime.

Spreading a single very long sequence across several GPUs needs one more idea. Ordinary attention has to ship a growing block of keys and values between GPUs as the context lengthens; linear attention only passes its fixed-size state, which is far cheaper. The complication is that KDA’s gated update won’t let you simply add up each GPU’s locally computed state, because its delta rule multiplies the incoming state by a token-dependent matrix, so what a segment does depends on the state that entered it. Moonshot’s answer splits each segment into two things a GPU can compute on its own, blind to the incoming state: a transition matrix that captures what the segment does to any state fed in, and a separate state built as if it had started from zero. Those pieces combine in order, so every GPU’s true starting state is rebuilt with a single fixed-size exchange and a scan down the ranks. The attention side of a million-token training step then needs only that small, fixed exchange per chunk, not traffic that grows with the sequence. The experts still pay their usual per-token shuffle between GPUs, but that was never the part that scaled with context — this fixes the part that did.

Most of the Work Is Building Environments

The closed labs sum up this stage in one line: “We trained it with reinforcement learning on agentic tasks.” Moonshot spends twenty pages on what that involves, and it’s the clearest view I’ve had of it.

Nearly all of K3’s reinforcement learning runs against environments the team had to build by hand, each one able to check its own answers. For coding there’s a GPU-kernel suite that scores a solution on correctness and speed against an expert version, wired up with a hacking detector that docks the model for faking the win through tricks like CUDA graph replay or dropping precision. For assistant work there are mock Gmail, Notion, Slack, and Canvas environments that keep state across simulated days, where one task can run to thousands of tool calls.

A couple are worth a closer look. The autonomous-execution tasks are the hardest of the set. Each one hands the agent a starting state, a goal, a set of tools, a budget, and a verifier — and the model must plan, execute, and recover from failures across long multi-step horizons. Building environments robust enough to train against at scale, without reward hacking or distributional collapse, turns out to be most of the engineering challenge. The architecture decisions described earlier set the ceiling for what the model can do; these environments are what determine whether it actually gets there.