Training a model is only half the work. Once it is running on your own GPUs, three techniques decide how fast it answers and how many people it can serve at once: the KV cache, PagedAttention and speculative decoding. None of them changes what the model says. They change how much memory and time each answer costs.
1. The KV cache: never redo finished work
A language model writes one token at a time, and every new token attends to all the tokens before it. Done naively, generating token 1,000 would mean recomputing the keys and values of the 999 tokens before it, even though none of them has changed. The KV cache stores those keys and values once and reuses them, so each step computes only what is new: the query, key and value of the latest token. The work per token stops growing with everything that came before, and generation becomes several times faster.
The price is memory. For every token in every conversation, each layer keeps a key and a value vector. The size is simple to estimate:
KV cache per token = 2 × layers × key-value heads × head dimension × bytes per value
Take a 70B-class model that uses grouped-query attention, with 80 layers and 8 key-value heads of dimension 128, stored in 16-bit precision. That is about 320 KB per token, or roughly 2.6 GB for a single 8,000-token conversation. Sixteen such conversations at once need around 42 GB for the cache alone, on top of the model weights. Grouped-query and multi-query attention exist precisely to shrink this: several attention heads share one set of keys and values.
The cache only matters at inference. During training the whole sequence is processed in parallel, so there is nothing to reuse.
2. PagedAttention: manage GPU memory like an operating system
Knowing the cache size is not enough; it also has to be placed in GPU memory. Traditional serving reserves one contiguous block per request, sized for the longest answer it might produce. Most requests finish far shorter, so the reserved space sits idle, and as requests come and go, free memory breaks into gaps too small for the next request. A request can be rejected while plenty of memory is technically free. The authors of vLLM (Kwon et al., SOSP 2023) found that existing serving systems wasted 60–80% of their KV-cache memory this way.
PagedAttention, the core of the open-source vLLM engine, borrows virtual memory from operating systems:
- Fixed-size blocks. The cache is cut into small blocks, commonly 16 tokens each, placed anywhere in GPU memory. A block table maps each conversation’s logical order to the physical blocks.
- Allocation on demand. A conversation takes a new block only when it fills the last one, and returns all its blocks the moment it finishes.
- Prefix sharing. When many requests start with the same text, such as a long system prompt shared by every user of an assistant, that prefix is stored once and referenced by all of them.
- Continuous batching. Instead of waiting for a whole batch to finish, a finished request is replaced by a waiting one at the very next step, so the GPU never idles.
In the vLLM paper, waste fell below 4%, and throughput rose 2–4× over earlier serving systems such as FasterTransformer and Orca at the same latency.
3. Speculative decoding: let a small model guess, let the big model check
Even with a perfect cache, each token still needs a full pass through the large model, one after another. Yet many tokens are easy: after “The capital of France is”, almost any model would say “Paris”. Speculative decoding exploits this. A small, fast draft proposes several tokens; the large model checks all of them in a single forward pass; correct tokens are kept, and generation resumes from the first wrong one.
It is lossless. Each drafted token is accepted with a probability based on how likely the large model finds it, and a rejected token is resampled from the large model’s own distribution. The result is statistically identical to running the large model alone, just with fewer expensive passes.
There are three common ways to draft:
- A separate small model from the same family: no training needed, but it costs extra GPU memory.
- Small trained draft heads on top of the large model (EAGLE, Medusa): little extra memory, but some GPU hours of training.
- Self-speculation: the model drafts using only its first layers, then verifies with the full depth.
The original speculative-decoding papers from Google Research and DeepMind reported roughly 2–3× faster generation with unchanged output, and trained-draft methods such as EAGLE and Medusa report similar or larger gains. Acceptance is highest for predictable output such as code and structured data, and lower for creative writing. Speculative decoding pays off with small batches and people waiting for long answers. It adds little when the GPUs are already saturated with large batches, or for very short replies.
How the three fit together
The KV cache makes each token cheap. PagedAttention packs those caches so that far more conversations fit on the same GPUs. Speculative decoding shortens the wait inside each conversation. The first two raise throughput, the third cuts latency, and modern serving engines such as vLLM combine all three.
What this means when the GPUs are your own
At AMD Telecom we train models on our own in-house GPUs, and serving on hardware you own has the same constraint: the GPUs are fixed, so every gigabyte counts. A few habits make a large difference:
- Size the KV cache before choosing a model. Multiply the per-token cost by your context length and the number of concurrent users. Very often the cache, not the weights, sets the limit.
- Serve with a paged cache and continuous batching. Leaving 60–80% of cache memory unused means buying GPUs you do not need.
- Put shared instructions in one common prefix. Prefix sharing turns a long system prompt from a per-user cost into a one-off.
- Use speculative decoding for interactive, structured work, such as assistants, data extraction and code, and measure acceptance on your real traffic.
For the other big memory lever, reducing the weights themselves, see LLM quantization explained. For how newer model architectures shrink the cache itself, see hybrid attention. If you are planning an AI application or want a model trained on your data, see our AI Model Development and AI Model Training services, or talk to our experts.