PagedAttention and RadixAttention: How LLM KV Cache Management Works

Modern LLMs rely on quantization, pruning, distillation, and faster attention kernels, but production performance often depends most on KV cache management. As context windows grow, the cache consumes significant GPU memory, limiting concurrency, throughput, and latency. Two breakthroughs transformed this challenge: PagedAttention improves memory allocation, while RadixAttention enables efficient prefix reuse.

Together, these techniques make LLM serving faster and more memory-efficient. In this article, we examine how PagedAttention and RadixAttention work, why they matter, and how they enable high-performance LLM serving.

Why the KV Cache Is the Real Bottleneck

Every transformer generates text one token at a time. For each new token, the model must attend to all previously generated tokens by using their key (K) and value (V) vectors. Recomputing these vectors at every step would make generation prohibitively expensive, so serving engines store them in memory as the KV cache. This cache eliminates redundant computation and makes autoregressive decoding practical, but it introduces a new challenge: memory consumption grows linearly with sequence length. For long-context models, the KV cache often becomes the largest dynamic consumer of GPU memory, determining how many requests can run simultaneously.

Why memory becomes the limiting factor

The size of the KV cache depends on the model architecture and the number of tokens stored. The per-token memory requirement is:

Where:

Symbol	Meaning
L	Number of transformer layers
Hkv	Number of KV heads
D	Head dimension
B	Bytes per value (2 for FP16)

For a Llama-3 8B class model with 32 layers, 8 KV heads, 128-dimensional heads, and FP16 precision, each token occupies approximately 128 KiB of KV cache. A 100,000-token context therefore requires nearly 12.8 GiB of memory before considering batching or additional requests.

The two fundamental problems

As GPU memory fills with KV tensors, serving systems encounter two distinct bottlenecks:

• Memory fragmentation: This occurs when the system allocates KV memory inefficiently, leaving large portions of GPU memory unusable and reducing the number of concurrent requests.
• Redundant computation: Identical prompt prefixes are repeatedly prefetched and encoded, even though their KV states have already been computed.

These problems are independent, and each inspired a different solution. PagedAttention addresses efficient memory allocation, while RadixAttention focuses on reusing previously computed KV cache across requests. Together, they define the foundation of modern LLM serving.

PagedAttention: Solving the Memory Allocation Problem

By 2023, the industry identified the biggest inefficiency in LLM serving as the storage method of the KV cache rather than attention itself. The system allocated one large contiguous block of GPU memory to hold the entire KV cache for every request. Since the serving engine could not predict how long a response would be, it typically reserved space close to the model’s maximum context length. Most of that memory remained unused throughout the request, drastically reducing the number of sequences that could be served simultaneously.

The problem with contiguous allocation

Traditional allocation creates two forms of fragmentation:

• Internal fragmentation: A request reserves thousands of token slots but generates only a small response, leaving most of the allocated memory idle.
• External fragmentation: As requests of different lengths finish, scattered gaps appear across GPU memory. Although the total free memory may be sufficient, it is no longer available as one contiguous block for new requests.

The result is poor GPU utilization and lower throughput, even when plenty of memory technically remains available.

How PagedAttention Works

The core idea behind PagedAttention is simple: allocate KV memory only when it is needed. Instead of reserving one large contiguous buffer for an entire sequence, the system divides the KV cache into fixed-size blocks (typically 16 or 32 tokens). As generation progresses, new blocks are allocated only after the previous one becomes full, allowing memory to grow incrementally rather than being over-provisioned from the start.

Step 1: Divide the KV cache into blocks

The system splits each sequence into equal-sized logical blocks, while the actual blocks can be stored anywhere in GPU memory.

Step 2: Use a block table for address translation

Every request maintains a block table that maps logical block IDs to their physical locations in GPU memory. During attention, the kernel consults this table to gather the required keys and values, making the sequence appear continuous even though its data is physically scattered.

Logical block	Physical GPU block
Block 0	Memory Block 18
Block 1	Memory Block 42
Block 2	Memory Block 07
Block 3	Memory Block 31

This indirection draws inspiration from page tables in operating systems: the model operates on a logical sequence, while the serving engine manages physical placement.

Step 3: Grow memory on demand

Instead of allocating space for thousands of future tokens, PagedAttention expands the KV cache one block at a time. A request generating 60 tokens occupies only the blocks required for those 60 tokens. No memory is reserved for tokens that may never be produced, which dramatically reduces internal fragmentation.

One of the most powerful features of PagedAttention is block sharing. If multiple requests begin with the same prompt, they reference the same physical KV blocks instead of storing duplicate tensors. When two requests eventually diverge, the system copies the shared block only at the point of modification, a mechanism known as copy-on-write. This makes prefix sharing highly memory-efficient for beam search, parallel sampling, and concurrent requests with identical system prompts.

Why this changed LLM serving

PagedAttention does not change the attention algorithm or the model’s outputs. Its innovation is purely architectural: it replaces inefficient contiguous allocation with a paged memory layout. The result is dramatically lower memory waste, higher GPU utilization, and the ability to serve many more concurrent requests on the same hardware.

RadixAttention: Solving the Prefix Reuse Problem

PagedAttention made GPU memory efficient, but it left another major inefficiency untouched: the system still recomputed identical prefixes for every new request. In real production workloads, requests are rarely independent. Thousands of users share the same system prompt, chat conversations repeatedly include their entire history, and agent workflows continuously append to an existing context. Consequently, the system spends much of the expensive prefill phase generating KV tensors that already exist.

RadixAttention eliminates this redundant computation by turning the KV cache into a searchable, reusable index rather than a temporary memory buffer.

The key idea: Store prefixes in a radix tree

Instead of discarding KV tensors when a request finishes, RadixAttention retains them inside a radix tree — a compressed trie where each edge represents a sequence of tokens. The system stores every unique prompt prefix once, while different requests branch only where their tokens begin to differ.

For example, three requests may begin with the same system prompt:

System: You are a helpful assistant.
User: What is AI?

System: You are a helpful assistant.
User: What is Machine Learning?

System: You are a helpful assistant.
User: What is Deep Learning?

Rather than storing three identical copies of the shared prefix, the radix tree keeps it once and creates separate branches only for the final user query.

How prefix matching works

When a new request arrives, RadixAttention performs three operations:

1. Match: Find the longest token prefix already present in the radix tree.
2. Reuse: Load the existing KV tensors for that matched prefix instead of recomputing them.
3. Insert: Compute only the unmatched suffix and append it back into the tree for future requests.

The longer the shared prefix, the less work the model performs during prefill. This directly reduces Time to First Token (TTFT), especially for long conversations and agentic applications.

PagedAttention vs. RadixAttention: What’s the Difference?

PagedAttention and RadixAttention solve different layers of the same underlying problem. PagedAttention addresses how memory is physically allocated — replacing large contiguous buffers with fixed-size paged blocks to eliminate fragmentation and enable on-demand growth. RadixAttention addresses what is stored in that memory — retaining computed KV tensors in a radix tree so that shared prompt prefixes are never recomputed across requests. In practice, the two techniques are complementary: PagedAttention provides the efficient memory substrate, while RadixAttention builds reusable prefix caching on top of it, together enabling the high-throughput, low-latency LLM serving that modern production deployments require.

Conclusion

KV cache management is the central bottleneck in production LLM serving. PagedAttention eliminates memory fragmentation by replacing contiguous allocation with a paged block system, dramatically improving GPU utilization and concurrent request capacity. RadixAttention eliminates redundant computation by storing computed KV tensors in a radix tree, allowing shared prompt prefixes to be reused across requests and reducing Time to First Token. Together, these two techniques form the architectural foundation of modern high-performance LLM inference engines, enabling faster, more efficient serving without changing model architecture or output quality.

Frequently Asked Questions

Q: Does PagedAttention change how attention is computed mathematically? No. PagedAttention only changes how KV tensors are stored and addressed in memory. The attention computation itself remains identical.

Q: Can PagedAttention and RadixAttention be used together? Yes. They are complementary. PagedAttention provides efficient physical memory management, and RadixAttention builds prefix reuse on top of that memory substrate.

Q: What workloads benefit most from RadixAttention? Workloads with high prefix overlap benefit most, including multi-turn chat applications, agentic pipelines, and deployments where many users share the same system prompt.

Q: What is copy-on-write in the context of PagedAttention? When multiple requests share a physical KV block and one request needs to modify it, the system copies that block before writing, preserving the shared state for other requests.