Look inside several of the newest long-context language models and one pattern stands out. In Alibaba’s Qwen3-Next, only one in every four sequence-mixing layers uses full attention; the three layers in between use a cheaper recurrent mixer. Moonshot AI’s Kimi Linear repeats the same 3:1 pattern, and Ant Group’s Ling 2.5 goes further, with seven cheap layers for every full-attention layer. Attention has not been removed. It has become selective.
This article explains why that is happening, how the cheap layers manage to remember, and what it means for teams that train and serve models on their own hardware.
The bill hybrids are cutting: the KV cache
Full (softmax) attention is what lets a transformer answer “what was the access code?” thousands of tokens later. The keys and values of every past token are kept in the KV cache, so a query can look up exactly the position it needs. The price is paid during generation: in every full-attention layer, each new token adds another set of keys and values, so the cache grows with the length of the context, multiplied by the number of layers, attention heads and head dimension. At a million tokens of context, the model reads gigabytes of cache for every token it writes.
Two trends have made this the main bottleneck. Reasoning models now write tens of thousands of output tokens to answer a single question, and agentic workflows run many rounds of tool calls that all share one long context. Anything that shrinks the KV cache is worth doing.
How the cheap layers remember
Linear attention replaces the growing cache with a fixed-size state: every token writes a small update into it, and a query reads from it. Cost now grows linearly with the length of the sequence instead of quadratically. The catch is that the naive version only ever adds. Once more associations have been written than the state can hold, they blur into each other, and precise recall fails.
The fixes arrived one layer at a time:
- The delta rule (DeltaNet). Before writing, the layer reads what the state currently holds for a key and writes only the difference, so an old association is replaced rather than piled on. Mathematically it is one step of online gradient descent on the memory for every token.
- A decay gate (from Mamba-2). Old information is allowed to fade, which gives the layer a learned memory horizon.
- Gated DeltaNet. Decay, targeted erase and write strength in one update. In the Gated DeltaNet paper (Yang, Kautz and Hatamizadeh, ICLR 2025), the combination outperformed both Mamba-2 and DeltaNet on in-context retrieval and long-context tasks.
- Per-channel decay (KDA, used in Kimi Linear). Every feature gets its own forgetting rate, so something like an identifier can persist while filler fades quickly.
Chunked formulations built from dense matrix multiplications let these layers train efficiently on GPUs.
Why keep any full attention at all?
A compressed memory is good, but it is not exact. When a task needs one precise detail from far back in the context, a few full-attention layers still do what a fixed-size state cannot, and that is why the 2026 hybrids keep a handful of them. Where they sit matters too: NVIDIA’s Jet-Nemotron paper searched which layers of a pre-trained model to keep as full attention and found that only a few specific layers need it for retrieval, while the rest can become linear or sliding-window layers.
The retrofitting trap
MiniMax shipped hybrid models in 2025 and then returned to plain full attention for MiniMax M2. In its own explanation, the company says its hybrid models matched full attention on standard benchmarks but showed clear deficits in complex multi-hop reasoning at larger scale. Variants it built by continuing to pre-train with cheaper attention, for hundreds of billions of tokens and more, lost ground on retrieval, multi-hop reasoning and in-context learning, and standard benchmarks were slow to reveal it.
The hybrids that are working in 2026, such as Qwen3-Next, Kimi Linear and Ling 2.5, were designed as hybrids from the start.
The payoff
In a hybrid model only the full-attention layers keep a per-token cache: about a quarter of the layers in a 3:1 design, and one in eight in a 7:1 design. Moonshot AI reports that Kimi Linear needs up to 75% less KV-cache memory than full attention and decodes up to six times faster at a context of one million tokens. The cache still grows with the context; what changed is how many layers pay for it.
The 2026 landscape
- Hybrid linear attention: Qwen3-Next, Qwen3.5, Kimi Linear, Ling 2.5. Mostly cheap layers, a few full-attention layers, designed in from scratch.
- Hybrid state-space and transformer: NVIDIA Nemotron 3, AI21’s Jamba, IBM Granite 4. The same idea, with state-space (Mamba) layers as the cheap mixer.
- Sparse attention: DeepSeek V3.2 and GLM-5, both using DeepSeek Sparse Attention. Attention in every layer, but each token attends only to a small, selected subset of earlier tokens.
- Full-attention holdouts: such as MiniMax M2.
The design question has moved from “attention or recurrence?” to “where in the layer stack is full attention worth paying for?”
What this means when you train and serve models on your own GPUs
At AMD Telecom we train and fine-tune models on our own in-house GPUs, with data collected by our BigParser platform and prepared on our big-data platform. For teams in the same position, the move to hybrid attention has three practical consequences:
- Memory decides context length and concurrency. On a fixed set of GPUs, KV-cache memory often limits how long a context you can serve and how many sessions run at once. Hybrid open-weight models can fit much longer contexts, or more concurrent users, on the same hardware.
- Fine-tune a hybrid; don’t build one by conversion. Start from a model that was designed as a hybrid, rather than retraining a full-attention model into one, which is the trap described above.
- Evaluate on your real workload. Throughput numbers and standard long-context benchmarks can look good while multi-step and agentic quality quietly drops. Test on the conversations and tool chains your application actually runs.
For the other levers that decide how much a model costs to run on your own hardware, see serving LLMs on your own GPUs and LLM quantization explained. If you are planning an AI application or want a model trained on your own data, see our AI Model Training and AI Model Development services, or talk to our experts.