Over the past few years most open-weight models have converged on the same answer: a gated feed-forward block built from SiLU, usually written SwiGLU, instead of a single GELU or ReLU in the middle of two projections. The switch adds a third weight matrix, which sounds like a 50% increase in feed-forward cost. It is not, because the hidden width is shrunk to compensate. This article covers where the activation sits, what gating actually buys you, the arithmetic behind the width rule, and the ways the change fails in practice.

Where the activation function lives

A transformer block does two things in sequence. Attention mixes information across tokens. The feed-forward network then processes each token position independently: project the hidden vector up to a wider space, apply a nonlinearity, project back down. No token talks to another inside it. It is a per-position feature transformation, and it is the main place in the block where a per-channel nonlinearity acts on the representation.

The parameter arithmetic is worth doing once. With model dimension d, the attention projections for queries, keys, values and output are four d x d matrices, so about 4d² weights. A classical feed-forward block with hidden width 4d uses two matrices of size d x 4d, so 8d² weights. That is two-thirds of the block’s parameters sitting in the feed-forward path, and a proportional share of its matrix-multiply FLOPs per token. Anything you change there is a budget decision.

What a single curve cannot do

ReLU zeroes negative inputs. GELU and SiLU are smooth alternatives that let a small amount of negative signal through near the origin and flatten out for large negative inputs; neither is strictly monotonic, but both behave like a soft, fixed bend. The important property they share is more basic than their shape: they are applied coordinate-wise. Each hidden unit’s output depends only on its own pre-activation value. The same curve is applied to every one of the thousands of channels, identically, at every position.

That means the block has no mechanism to decide, based on the input, that channel 417 should be suppressed in this context while channel 93 should pass through at full strength. The weights can scale channels, but the scaling is fixed after training. Everything the nonlinearity contributes is a static bend, not a routing decision.

Gated linear units: a gate path and a value path

Gated linear units, introduced for convolutional language models by Dauphin et al. (2017), change that. Instead of one up-projection, you compute two from the same input: a value vector and a gate vector. The gate is passed through a squashing nonlinearity and multiplied element-wise into the value. The product is then projected back down to the model dimension.

The consequence is multiplicative rather than additive. Each output channel is now a product of two different projections of the input, one of them passed through a nonlinearity, so a channel can be driven towards zero by evidence that has nothing to do with its own value path. In practical terms:

• The block can open some feature lanes and close others depending on the token and its context.
• Interactions between channels become second-order, not just the first-order mixing a linear layer provides.
• Gradients flow through two paths, which changes the conditioning of training — usually for the better, sometimes for the worse if the gate degenerates.

SwiGLU: SiLU on the gate, linear on the value

SwiGLU is the variant that stuck. The gate uses SiLU, also known as swish, defined as x * sigmoid(x): smooth, unbounded above, with no flat dead region for small negative inputs. The value path stays linear. The full block is down(SiLU(x @ W_gate) * (x @ W_up)), with * as an element-wise product. Biases are typically omitted.

The systematic comparison comes from Shazeer’s GLU Variants Improve Transformer (2020), which evaluates several gated feed-forward variants against ReLU and GELU baselines on text-to-text pretraining and reports better perplexity for the gated variants at matched parameter counts. The PaLM paper (Chowdhery et al., 2022) and Meta’s LLaMA paper (Touvron et al., 2023) both state that they use SwiGLU in their feed-forward layers, and most open-weight model families released since have followed.

The 8/3 rule, and why it is not optional

Three matrices instead of two is the obvious objection. The fix is to narrow the hidden dimension. Call the gated hidden width h. The gated block uses three matrices between d and h, so 3dh parameters. Setting that equal to the 8d² of the classical block gives h = 8d/3 — two-thirds of the usual 4d. Parameter count matches, and since feed-forward FLOPs per token scale with parameter count, so does compute.

LLaMA 7B is a clean illustration: the LLaMA paper (Touvron et al., 2023) gives it a model dimension of 4096, and the released model configuration sets the feed-forward hidden size to 11008. Eight thirds of 4096 is about 10,923; 11008 is that value rounded up to a multiple of 256, which keeps the matrix shapes friendly to GPU kernels.

The mistake to avoid is keeping 4d while adding the gate. You will get a model that trains better than the baseline and conclude SwiGLU works, when in fact you have simply spent 50% more on the feed-forward block. Any comparison worth acting on is run at matched parameters and matched FLOPs, not matched width.

Implementation details that decide whether it works

• Fuse the two up-projections. Materialise a single matrix of width 2h and split the result into gate and value halves. One larger matrix multiply generally uses the hardware better than two smaller ones.
• Round the hidden width. Take 8d/3 as a target, then round to a multiple that suits your kernels and your tensor-parallel degree, as LLaMA does.
• Put dropout after the product. Dropping units inside the gate path corrupts the gating signal itself rather than regularising the block’s output.
• Watch the gate path at initialisation. If the gate collapses towards a constant early in training, the block degenerates into a plain projection-plus-nonlinearity and the benefit disappears. Initialisation scale and learning rate for that projection are the first things to check when a gated model underperforms.
• Do not assume the gain transfers. The reported improvements come from language-model pretraining. On small models, narrow domains or non-language tasks, the difference can be within noise, and you need your own matched-budget comparison to know.

What this means for organisations training their own models

The useful lesson is not “use SwiGLU” — most frameworks already default to it. It is that the strongest architecture decisions are the budget-neutral ones: a change that improves loss while holding parameters and FLOPs fixed compounds across every training run and every deployment of that model. Evaluating such a change honestly requires discipline about the control, which is why the 8/3 width rule matters as much as the gating formula. The same reasoning applies elsewhere in the stack, from how much full attention a long-context model actually needs to how many bits each weight is stored in at serving time.

It also affects planning. A gated feed-forward block changes the shape of every matrix in two-thirds of your parameter budget, which propagates into memory layout, tensor-parallel splits and checkpoint sizes. Decide it before you start a long pretraining or continued-pretraining run, not after. AMD Telecom trains and fine-tunes deep-learning models and large language models on its own in-house GPUs.

If you are weighing architecture choices for a model you intend to train on your own data, our AI Model Training and AI Model Development pages cover this work. Contact us to discuss your case.

The post SwiGLU Explained: Why Modern LLMs Replaced GELU in the Feed-Forward Block appeared first on AMD Telecom.

FlashAttention, introduced in 2022 by Dao et al. as FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, is built around that observation. It does not approximate attention, drop tokens or sparsify the score matrix. It reorganises the computation so that the large intermediate matrices are never written to main memory at all: the same output, far less data movement. Variants of it are now the default attention path in mainstream training and serving stacks.

Attention is memory-bound, not compute-bound

Consider what a single attention head does for a sequence of length N. It computes an N × N matrix of scores, applies a softmax across each row, then multiplies the resulting probabilities by the values. A naive implementation materialises two full N × N matrices — the raw scores and the post-softmax probabilities — in memory.

The arithmetic is worth doing yourself. At a sequence length of 8,192 in 16-bit precision, one such matrix is 8,192 × 8,192 × 2 bytes ≈ 134 MB — for a single head. With 32 heads that is about 4 GB of intermediate data for one attention layer in one forward pass, all of it written out and read back, with the softmax step touching it twice.

Meanwhile the matrix multiplications are exactly the dense, regular work that tensor cores handle well. Arithmetic throughput has grown faster than memory bandwidth, so the kernel spends most of its wall-clock time waiting for transfers. Adding FLOPs to such a kernel costs almost nothing; removing memory traffic is what makes it faster.

Two levels of memory, one bottleneck

The hardware picture is simple. High-bandwidth memory (HBM) is the large pool quoted on the spec sheet, tens of gigabytes. On-chip SRAM — the shared memory and registers attached to each streaming multiprocessor — is tiny by comparison: the FlashAttention paper puts an A100 at 40 GB of HBM against roughly 20 MB of on-chip SRAM in total, with the on-chip memory roughly an order of magnitude faster.

An N × N score matrix at long context does not fit in 20 MB, so a standard implementation has no choice but to stage it through HBM, and every round trip is a stall. The design question becomes: can we produce the same result while keeping every intermediate inside the fast memory?

Three ideas that make it work

• Tiling. Split the queries, keys and values into blocks small enough that a tile of the score matrix fits in SRAM. Load a block of queries, stream blocks of keys and values past it, and compute the scores, their softmax contribution and the output accumulation inside fast memory. Q, K and V are read once; the output is written once.
• Online softmax. Softmax normally needs the whole row: subtract the row maximum for numerical stability, then divide by the sum of exponentials. Tiling means you only ever see part of a row. The fix is to carry a running maximum and a running sum alongside the partial output, rescaling the accumulated result whenever a new tile raises the maximum. The algebra works out exactly.
• Recomputation. The backward pass needs the attention probabilities to compute gradients. Rather than keeping them from the forward pass — which would reintroduce the memory cost — FlashAttention recomputes them from Q and K using the saved softmax statistics. That is more arithmetic in exchange for less traffic, the right trade when arithmetic is the abundant resource.

Put together, the kernel is an outer loop over query blocks and an inner loop over key and value blocks, with the running statistics and the partial output held in fast memory. The output is exact — the same attention, not a cheaper approximation of it — which is what makes it safe to apply to an already-trained model.

What changed in versions 2 and 3

The first version parallelised across the batch and the attention heads, which leaves GPUs underused when the batch is small and the sequence is long — precisely the long-context regime people care about. FlashAttention-2 (Dao, 2023) added parallelism over the sequence dimension, reduced the non-matmul work around the softmax and improved how work is partitioned between warps. The paper reports roughly 2× over the original, with attention reaching 50–73% of the theoretical maximum matmul throughput on an A100.

FlashAttention-3 (Shah et al., 2024) targets Hopper-class hardware, exploiting asynchronous tensor-core instructions and the tensor memory accelerator through warp specialisation: some warps do nothing but fetch the next tile while others compute on the current one, forming a producer–consumer pipeline. The paper reports 1.5–2.0× over FlashAttention-2 on H100 in FP16, and higher throughput again in FP8.

Each generation extracts more overlap between data movement and arithmetic, and each is tied more tightly to a particular GPU architecture. Related work such as ring attention extends the same tiling logic across devices, splitting one sequence over several GPUs.

Turning it on, and when you do not get it

In practice this is rarely a code-writing exercise. PyTorch dispatches scaled_dot_product_attention to a fused backend when the inputs and hardware allow it, and Hugging Face Transformers exposes an attention implementation argument when a model is loaded. The work is in the conditions, not the call:

• Hardware generation. The flash-attn project requires Ampere or newer for FlashAttention-2, and its version 3 kernels target Hopper. Older cards fall back to a slower path.
• Dtype and shape constraints. Fused kernels typically require half or bfloat16 inputs and particular head dimensions. Step outside them and you silently get the reference implementation.
• Masking and bias. Causal and padding masks are supported; arbitrary additive attention biases often are not, and a custom mask is a common reason a model quietly loses the fast path.
• Verify, do not assume. Profile the kernel that actually ran. A change upstream can disable fusion without any error message.

It also composes with the rest of the stack. Memory saved on activations is memory available for the KV cache, which is why these kernels and the paged cache described in serving LLMs on your own GPUs are usually deployed together, and why hybrid attention designs assume an efficient exact kernel as their baseline.

What this means for organisations training on their own hardware

The lesson generalises beyond attention. When you budget a training run or a fine-tune, the first question is not how many FLOPs the model needs but where the bytes are going; a kernel waiting on HBM will not go faster on a card with more tensor cores. Before reaching for architectural changes that alter model quality, confirm that the exact, well-optimised kernels are actually being used — throughput with no accuracy trade-off, and often the difference between a long-context experiment being affordable or not.

It also changes how you read hardware requirements. Because these kernels are tied to specific GPU generations and precisions, the gap between an older and a newer card is wider than the headline specifications suggest. AMD Telecom trains and fine-tunes deep-learning models and large language models on its own in-house GPUs.

If you are planning a training run or assessing what your infrastructure can realistically support, our AI Model Training and AI Model Development services are a good place to start — contact us to discuss your requirements.

The post Why Attention Is Memory-Bound: Inside FlashAttention and IO-Aware Kernels appeared first on AMD Telecom.

From likely text to correct answers

Pre-training teaches a model to predict the next token, so it is rewarded for sounding likely, not for being right. Instruction tuning and preference training make it fluent, helpful and safe, but none of these stages directly rewards multi-step reasoning. Reasoning models add a further stage: reinforcement learning on problems whose answers can be checked. The model writes a chain of thought and an answer; a correct answer earns a reward, a wrong one does not. Nobody tells it how to reason.

The classic recipe needed four networks

Reinforcement learning from human feedback (RLHF) runs four models side by side:

• the policy, which writes the answers;
• a reward model, trained on thousands of human rankings to score them;
• a critic (value model), which estimates a baseline so each update knows whether an answer did better or worse than expected;
• a frozen reference copy, which stops the policy drifting too far.

Two of the four, the reward model and the critic, exist only to manufacture a learning signal.

Verifiable rewards remove the reward model

For maths and code there is no need to learn what a good answer looks like: you can check it. Run the tests, compare the final number, verify the proof. This is RLVR, reinforcement learning with verifiable rewards: a verifier returns 1 or 0, with no human labels and no training. The reward is still there; only the learning of it is gone. The boundary is honest: it works where cheap, exact checks exist, gets expensive for proofs and agentic tasks, and does not apply to open-ended writing.

GRPO removes the critic

GRPO (group relative policy optimisation) replaces the critic with something almost embarrassingly simple. For each prompt, sample a group of answers and score each one. The group’s average reward is the baseline; an answer’s advantage is how far it landed above or below its peers. The update itself uses the same clipped objective as PPO. Four models become two: the policy and the reference. That means one less large network in GPU memory during training.

What emerged when DeepSeek let it run

DeepSeek-R1-Zero skipped the usual warm-up of supervised examples entirely and applied RLVR straight to a pre-trained base model. Nobody rewarded length, yet the average response length grew steadily during training, because longer reasoning reached verifiable answers more often. The DeepSeek-R1 paper (2025) describes behaviours such as re-checking and reflecting on earlier steps emerging without any demonstrations. The outputs were hard to read, sometimes switching language mid-thought, so the full DeepSeek-R1 used four stages instead: a small supervised “cold start” for readable style, then RL, then fine-tuning on the model’s own best answers, then RL again. A strong reasoning model can then be distilled: smaller models fine-tuned on its reasoning traces gain much of the ability without any RL of their own.

What goes wrong in practice

• Entropy collapse. Rewards concentrate probability on what already works. Push too far and the model stops exploring, and the reward curve flattens because it stopped trying anything new, not because it solved the task.
• Hidden biases in the objective. Dividing by response length dilutes the signal for long correct answers, which quietly teaches the model that longer is better. Dividing by the group’s standard deviation over-weights problems that are trivially easy or impossibly hard. The Dr. GRPO paper (“Understanding R1-Zero-Like Training”, 2025) simply drops both terms.
• Mixture-of-experts instability. In sparse models, the router’s choice of experts changes between updates. The Qwen team’s GSPO paper (2025) reports that about 10% of the activated experts change after each update in Qwen3-30B-A3B. Token-level probability ratios then explode or collapse. GSPO computes the ratio over the whole sequence instead, which averages out the churn; the Qwen team used it to train its Qwen3 models.

Two caveats worth knowing

Does RL create new reasoning, or amplify what was already there? The evidence points both ways. A 2025 study from Tsinghua University (“Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?”) found that a base model sampled enough times eventually matches the RL-trained one; other studies on controlled tasks report abilities the base model never shows. The question is open.

Reasoning is not factual reliability. RL rewards correctness where it can be verified; open-ended facts get no such feedback. OpenAI’s own system card for o3 reports a hallucination rate of 33% on its PersonQA benchmark, against 16% for o1. A model can reason confidently when it should be uncertain, so factual answers still need grounding in your own data.

What this means for training models on your own data

At AMD Telecom we train models on our own in-house GPUs. RL with verifiable rewards is interesting well beyond maths contests, because many business tasks can be checked automatically:

• data extracted from a document compared against the system of record;
• generated SQL that must run and return the expected rows;
• classifications with known labels;
• structured output that must satisfy a schema (see how LLMs are forced to return valid JSON).

Where such a verifier exists, critic-free methods like GRPO make the training loop simpler and lighter on GPU memory. Where it does not, you are back to human judgement, and your evaluation should be as careful as your training.

If you want a model trained on your own data and checked against your own definition of correct, see our AI Model Training and AI Model Development services, or talk to our experts.

The post How Reasoning LLMs Are Trained: Verifiable Rewards and GRPO appeared first on AMD Telecom.

Why prompting and retries are not enough

The obvious first step is to ask for JSON in the prompt. That makes the right structure more likely, but it cannot make every wrong continuation impossible. The usual defence is a pipeline: generate, parse, validate against a schema, and retry or repair on failure. Take a simple weather lookup to see its limits. One attempt drops a quote and fails the parser. Another is valid JSON but writes units instead of unit, so it fails the schema. Both are caught, but only after the tokens were generated and paid for. Repair can fix punctuation, but it becomes guesswork once the mistake is bigger than a comma.

Constrained decoding: block illegal tokens before they are chosen

At every step the model scores thousands of candidate next tokens, and a sampler picks one. Constrained (grammar-based) decoding places a filter between those two steps. Any token that would make a valid completion impossible is masked: its probability is set to zero. The sampler can only choose among the tokens that survive. Legal tokens keep their relative scores; the mask does not make them equally likely and does not change the model’s weights.

The loop is mask, sample, advance. After each token the constraint moves on, because what is legal next depends on everything generated so far. The model is not even allowed to stop early: the end-of-output token is masked until the object is complete. The same idea works for other formal languages, such as SQL, XML or program syntax, not just JSON.

The hard part: tokens are not characters

A model token can contain a quote, a comma, part of a field name, or several JSON boundaries at once. Suppose the output so far is {"city": "Paris", "unit": " and the only allowed values are C or F. The tokens C, C" and even C"} all keep a valid completion possible. Celsius starts with the right letter but cannot be completed validly, so it is masked, however highly the model scored it. A reliable constraint therefore tests the whole token against the grammar, not just its first character.

Simple choices such as C-or-F can be tracked with a small finite-state machine. Nested objects and arrays need parser state that remembers every bracket still to be closed. In practice an engine compiles the supported part of the JSON Schema, together with the model’s tokenizer, into a recognizer that can answer “which tokens are legal now?” quickly at every step.

How well does it work?

Three questions are easy to blur, and deserve separate answers:

• Validity. When OpenAI introduced strict structured outputs in 2024, it reported that its model matched the supplied JSON schemas in 100% of its internal tests, against under 40% for an earlier model relying on instructions alone.
• Coverage. Engines enforce different subsets of JSON Schema. The JSONSchemaBench study (2025) found marked differences in which schema features the popular engines support.
• Speed. Overhead depends on the engine, hardware, schema and caching. It can be negligible or significant, so measure your own setup.

What it guarantees, and what it does not

Constrained decoding guarantees form: every emitted token kept a valid completion possible, and output ended only when the structure was complete. It does not guarantee:

• Meaning. A call for Paris is perfectly valid even if the user asked about Prague.
• Completion. A token limit, a cancellation or a dropped connection can still cut the output short.
• Permission. A well-formed request is not an authorised one.

Those remain separate layers: semantic checks against the request, permission gates with least privilege, and validation of the effect in the downstream system. In short: the decoder can guarantee the brackets, not the weather, and not the permission to fetch it.

Why this matters for AI in business processes

Assistants that trigger actions and models that automate transactions and workflows, part of our AI Model Development service, live or die by machine-readable output. Constrained decoding removes a whole class of failures, such as broken JSON, missing fields and invented enum values, before they happen, instead of catching them with retries. It works best alongside:

• schemas kept deliberately simple and within what your engine fully supports;
• business-rule checks on the values themselves;
• explicit permissions on every action a model can trigger.

Structured, predictable output also combines well with speculative decoding for speed; see serving LLMs on your own GPUs. If you are planning an AI application that has to talk to your systems, see our AI Model Development and AI Model Training services, or talk to our experts.

The post How LLMs Are Forced to Return Valid JSON: Constrained Decoding Explained appeared first on AMD Telecom.

Smaller is also faster

It is natural to assume that generating text is limited by compute. For one conversation at a time, it is not: producing each token means reading every weight from GPU memory once, and the GPU’s arithmetic units spend most of their time waiting for data. Memory bandwidth is the real bottleneck, so fewer bytes per weight translate almost directly into more tokens per second on the same chip.

How quantization works

Quantization maps continuous values onto a fixed grid. An 8-bit integer has 256 levels, and a 4-bit one only 16. A scale sets how much real value each step represents, and a zero point makes sure zero is exactly representable. Each weight snaps to the nearest grid point; the gap is the quantization error. At 8 bits the grid is fine and errors are tiny; at 4 bits they become large enough that simple rounding is no longer good enough.

Granularity matters as much as bit width. One scale for a whole matrix is dominated by a few outlier values and wastes most of the grid on everyone else. Giving each row its own scale is better; giving each small group of weights (typically 128) its own scale is better still, at a small storage cost. Per-group scaling is the standard for 4-bit models.

The number formats differ too. BF16 keeps the wide range of 32-bit floats with less precision, which is why it is preferred for training, where gradients can spike. FP16 has more precision but overflows above 65,504. FP8 is becoming a training format in its own right. NF4 is a 4-bit lookup table whose 16 values follow a normal distribution, matching the shape of real weight distributions.

The main methods

Almost everything deployed today uses post-training quantization: take a trained model and a small calibration set of a few hundred samples, and quantize in minutes to hours instead of retraining for days. The best-known methods:

• GPTQ quantizes a layer one column at a time and adjusts the remaining weights to compensate for each rounding error, using second-order information about which weights the output is most sensitive to. The GPTQ paper (Frantar et al., 2022) showed 4-bit models staying close to their 16-bit accuracy where plain rounding degrades sharply, and quantized a 175-billion-parameter model in about four GPU hours.
• AWQ (activation-aware weight quantization) observes that about 1% of weight channels matter far more than the rest, judged by how strongly they are used, not by how large they are. It scales those channels up before quantizing so that they lose less precision. It won best paper at MLSys 2024.
• SmoothQuant targets 8-bit weights and 8-bit activations. It moves the difficulty of rare activation outliers into the weights, so that both can run on 8-bit hardware for high-throughput serving.
• GGUF, the format behind llama.cpp, runs quantized models on CPUs and consumer hardware. Its “K-quants” such as Q4_K_M give sensitive layers more bits and average a little under five bits per weight, so a 7B model shrinks from about 13.5 GB in 16-bit to around 4 GB. Q4_K_M is widely used as the default balance of size and quality.

Fine-tuning on limited GPUs: QLoRA

Quantization is not only for inference. QLoRA freezes the base model in 4-bit NF4 and trains only small low-rank adapter matrices kept in 16-bit, so gradients flow through a tiny fraction of the parameters. A second trick, double quantization, compresses the scale factors themselves. The result, as the QLoRA paper (Dettmers et al., 2023) showed, is that a 65B model can be fine-tuned on a single 48 GB GPU, which put large-model fine-tuning within reach of teams without data-centre budgets.

Don’t forget the KV cache

At long context lengths the KV cache, the stored keys and values of the conversation so far, can outgrow the weights. For a model like Llama 2 70B it is about 320 KB per token in 16-bit, or roughly 40 GB at 128,000 tokens. Storing the cache in 8 bits halves that memory and 4 bits quarters it. Serving engines such as vLLM support an 8-bit (FP8) cache; the quality cost is usually small but should be measured on your own tasks. Our article on serving LLMs on your own GPUs covers the cache in more detail.

How low can you go?

Larger models tolerate compression better, because their redundancy absorbs the rounding noise. Research on the trade-off between bits and accuracy (Dettmers and Zettlemoyer, “The case for 4-bit precision”, 2023) found 4-bit to be almost always the best choice for a fixed memory budget: a larger model at 4 bits beats a smaller model at higher precision. Below about 3 bits, standard methods break down. Research methods such as QuIP#, AQLM and QTIP use vector quantization to make 2-bit usable, and Microsoft’s BitNet b1.58 goes to three values per weight, but it has to be trained that way from scratch.

A practical map

• One user or a few, memory-bound: 4-bit weights with 16-bit activations (GPTQ or AWQ).
• High-throughput serving: 8-bit weights and activations, or FP8 on hardware that supports it.
• CPUs, laptops and edge devices: GGUF, starting from Q4_K_M.
• Fine-tuning on your own data with limited GPUs: QLoRA.

What this means when the GPUs are your own

At AMD Telecom we train models on our own in-house GPUs. On fixed hardware, quantization decides which model sizes fit, how many can run side by side and how many tokens per second each one delivers. It also makes adapting larger models to your own data practical. One caution applies everywhere: perplexity is only a proxy. Always evaluate the quantized model on your own tasks, such as your documents, your languages and your output formats, before it goes into production.

If you want a model trained or adapted on your own data, see our AI Model Training and AI Model Development services, or talk to our experts.

The post LLM Quantization Explained: Running Bigger Models on Fewer GPUs appeared first on AMD Telecom.

1. The KV cache: never redo finished work

A language model writes one token at a time, and every new token attends to all the tokens before it. Done naively, generating token 1,000 would mean recomputing the keys and values of the 999 tokens before it, even though none of them has changed. The KV cache stores those keys and values once and reuses them, so each step computes only what is new: the query, key and value of the latest token. The work per token stops growing with everything that came before, and generation becomes several times faster.

The price is memory. For every token in every conversation, each layer keeps a key and a value vector. The size is simple to estimate:

KV cache per token = 2 × layers × key-value heads × head dimension × bytes per value

Take a 70B-class model that uses grouped-query attention, with 80 layers and 8 key-value heads of dimension 128, stored in 16-bit precision. That is about 320 KB per token, or roughly 2.6 GB for a single 8,000-token conversation. Sixteen such conversations at once need around 42 GB for the cache alone, on top of the model weights. Grouped-query and multi-query attention exist precisely to shrink this: several attention heads share one set of keys and values.

The cache only matters at inference. During training the whole sequence is processed in parallel, so there is nothing to reuse.

2. PagedAttention: manage GPU memory like an operating system

Knowing the cache size is not enough; it also has to be placed in GPU memory. Traditional serving reserves one contiguous block per request, sized for the longest answer it might produce. Most requests finish far shorter, so the reserved space sits idle, and as requests come and go, free memory breaks into gaps too small for the next request. A request can be rejected while plenty of memory is technically free. The authors of vLLM (Kwon et al., SOSP 2023) found that existing serving systems wasted 60–80% of their KV-cache memory this way.

PagedAttention, the core of the open-source vLLM engine, borrows virtual memory from operating systems:

• Fixed-size blocks. The cache is cut into small blocks, commonly 16 tokens each, placed anywhere in GPU memory. A block table maps each conversation’s logical order to the physical blocks.
• Allocation on demand. A conversation takes a new block only when it fills the last one, and returns all its blocks the moment it finishes.
• Prefix sharing. When many requests start with the same text, such as a long system prompt shared by every user of an assistant, that prefix is stored once and referenced by all of them.
• Continuous batching. Instead of waiting for a whole batch to finish, a finished request is replaced by a waiting one at the very next step, so the GPU never idles.

In the vLLM paper, waste fell below 4%, and throughput rose 2–4× over earlier serving systems such as FasterTransformer and Orca at the same latency.

3. Speculative decoding: let a small model guess, let the big model check

Even with a perfect cache, each token still needs a full pass through the large model, one after another. Yet many tokens are easy: after “The capital of France is”, almost any model would say “Paris”. Speculative decoding exploits this. A small, fast draft proposes several tokens; the large model checks all of them in a single forward pass; correct tokens are kept, and generation resumes from the first wrong one.

It is lossless. Each drafted token is accepted with a probability based on how likely the large model finds it, and a rejected token is resampled from the large model’s own distribution. The result is statistically identical to running the large model alone, just with fewer expensive passes.

There are three common ways to draft:

• A separate small model from the same family: no training needed, but it costs extra GPU memory.
• Small trained draft heads on top of the large model (EAGLE, Medusa): little extra memory, but some GPU hours of training.
• Self-speculation: the model drafts using only its first layers, then verifies with the full depth.

The original speculative-decoding papers from Google Research and DeepMind reported roughly 2–3× faster generation with unchanged output, and trained-draft methods such as EAGLE and Medusa report similar or larger gains. Acceptance is highest for predictable output such as code and structured data, and lower for creative writing. Speculative decoding pays off with small batches and people waiting for long answers. It adds little when the GPUs are already saturated with large batches, or for very short replies.

How the three fit together

The KV cache makes each token cheap. PagedAttention packs those caches so that far more conversations fit on the same GPUs. Speculative decoding shortens the wait inside each conversation. The first two raise throughput, the third cuts latency, and modern serving engines such as vLLM combine all three.

What this means when the GPUs are your own

At AMD Telecom we train models on our own in-house GPUs, and serving on hardware you own has the same constraint: the GPUs are fixed, so every gigabyte counts. A few habits make a large difference:

• Size the KV cache before choosing a model. Multiply the per-token cost by your context length and the number of concurrent users. Very often the cache, not the weights, sets the limit.
• Serve with a paged cache and continuous batching. Leaving 60–80% of cache memory unused means buying GPUs you do not need.
• Put shared instructions in one common prefix. Prefix sharing turns a long system prompt from a per-user cost into a one-off.
• Use speculative decoding for interactive, structured work, such as assistants, data extraction and code, and measure acceptance on your real traffic.

For the other big memory lever, reducing the weights themselves, see LLM quantization explained. For how newer model architectures shrink the cache itself, see hybrid attention. If you are planning an AI application or want a model trained on your data, see our AI Model Development and AI Model Training services, or talk to our experts.

The post Serving LLMs on Your Own GPUs: KV Cache, PagedAttention and Speculative Decoding appeared first on AMD Telecom.

This article explains why that is happening, how the cheap layers manage to remember, and what it means for teams that train and serve models on their own hardware.

The bill hybrids are cutting: the KV cache

Full (softmax) attention is what lets a transformer answer “what was the access code?” thousands of tokens later. The keys and values of every past token are kept in the KV cache, so a query can look up exactly the position it needs. The price is paid during generation: in every full-attention layer, each new token adds another set of keys and values, so the cache grows with the length of the context, multiplied by the number of layers, attention heads and head dimension. At a million tokens of context, the model reads gigabytes of cache for every token it writes.

Two trends have made this the main bottleneck. Reasoning models now write tens of thousands of output tokens to answer a single question, and agentic workflows run many rounds of tool calls that all share one long context. Anything that shrinks the KV cache is worth doing.

How the cheap layers remember

Linear attention replaces the growing cache with a fixed-size state: every token writes a small update into it, and a query reads from it. Cost now grows linearly with the length of the sequence instead of quadratically. The catch is that the naive version only ever adds. Once more associations have been written than the state can hold, they blur into each other, and precise recall fails.

The fixes arrived one layer at a time:

• The delta rule (DeltaNet). Before writing, the layer reads what the state currently holds for a key and writes only the difference, so an old association is replaced rather than piled on. Mathematically it is one step of online gradient descent on the memory for every token.
• A decay gate (from Mamba-2). Old information is allowed to fade, which gives the layer a learned memory horizon.
• Gated DeltaNet. Decay, targeted erase and write strength in one update. In the Gated DeltaNet paper (Yang, Kautz and Hatamizadeh, ICLR 2025), the combination outperformed both Mamba-2 and DeltaNet on in-context retrieval and long-context tasks.
• Per-channel decay (KDA, used in Kimi Linear). Every feature gets its own forgetting rate, so something like an identifier can persist while filler fades quickly.

Chunked formulations built from dense matrix multiplications let these layers train efficiently on GPUs.

Why keep any full attention at all?

A compressed memory is good, but it is not exact. When a task needs one precise detail from far back in the context, a few full-attention layers still do what a fixed-size state cannot, and that is why the 2026 hybrids keep a handful of them. Where they sit matters too: NVIDIA’s Jet-Nemotron paper searched which layers of a pre-trained model to keep as full attention and found that only a few specific layers need it for retrieval, while the rest can become linear or sliding-window layers.

The retrofitting trap

MiniMax shipped hybrid models in 2025 and then returned to plain full attention for MiniMax M2. In its own explanation, the company says its hybrid models matched full attention on standard benchmarks but showed clear deficits in complex multi-hop reasoning at larger scale. Variants it built by continuing to pre-train with cheaper attention, for hundreds of billions of tokens and more, lost ground on retrieval, multi-hop reasoning and in-context learning, and standard benchmarks were slow to reveal it.

The hybrids that are working in 2026, such as Qwen3-Next, Kimi Linear and Ling 2.5, were designed as hybrids from the start.

The payoff

In a hybrid model only the full-attention layers keep a per-token cache: about a quarter of the layers in a 3:1 design, and one in eight in a 7:1 design. Moonshot AI reports that Kimi Linear needs up to 75% less KV-cache memory than full attention and decodes up to six times faster at a context of one million tokens. The cache still grows with the context; what changed is how many layers pay for it.

The 2026 landscape

• Hybrid linear attention: Qwen3-Next, Qwen3.5, Kimi Linear, Ling 2.5. Mostly cheap layers, a few full-attention layers, designed in from scratch.
• Hybrid state-space and transformer: NVIDIA Nemotron 3, AI21’s Jamba, IBM Granite 4. The same idea, with state-space (Mamba) layers as the cheap mixer.
• Sparse attention: DeepSeek V3.2 and GLM-5, both using DeepSeek Sparse Attention. Attention in every layer, but each token attends only to a small, selected subset of earlier tokens.
• Full-attention holdouts: such as MiniMax M2.

The design question has moved from “attention or recurrence?” to “where in the layer stack is full attention worth paying for?”

What this means when you train and serve models on your own GPUs

At AMD Telecom we train and fine-tune models on our own in-house GPUs, with data collected by our BigParser platform and prepared on our big-data platform. For teams in the same position, the move to hybrid attention has three practical consequences:

• Memory decides context length and concurrency. On a fixed set of GPUs, KV-cache memory often limits how long a context you can serve and how many sessions run at once. Hybrid open-weight models can fit much longer contexts, or more concurrent users, on the same hardware.
• Fine-tune a hybrid; don’t build one by conversion. Start from a model that was designed as a hybrid, rather than retraining a full-attention model into one, which is the trap described above.
• Evaluate on your real workload. Throughput numbers and standard long-context benchmarks can look good while multi-step and agentic quality quietly drops. Test on the conversations and tool chains your application actually runs.

For the other levers that decide how much a model costs to run on your own hardware, see serving LLMs on your own GPUs and LLM quantization explained. If you are planning an AI application or want a model trained on your own data, see our AI Model Training and AI Model Development services, or talk to our experts.

The post Hybrid Attention: Why 2026’s Long-Context LLMs Use Less Full Attention appeared first on AMD Telecom.