Featured image titled RoPE, Rotate Q and K, with three takeaways: rotate Q and K only, never the value vectors; scores depend on n minus m, not absolute positions; extend context with PI, NTK or YaRN, then test.

RoPE Explained: How Rotary Embeddings Give LLMs Word Order

Self-attention has no built-in notion of sequence. Shuffle a sentence’s tokens and a transformer block computes the same pairwise interactions, just reordered. Word order has to be injected deliberately. The method used by most current open-weight model families for that job is rotary position embedding, or RoPE. It was introduced in the RoFormer paper (Su et al., 2021) and adopted widely, including in Meta’s Llama series, although recent designs no longer apply it in every layer.

RoPE matters practically: it decides what happens when a model reads more text than it was trained on. Context-length extension, the scaling factors in a config file, the quality degradation that appears beyond the training window, the way positions are tracked across a KV cache — all of it comes back to how rotary embeddings are defined. This article covers the idea, the property that makes it work, and the decisions that matter when training or serving models.

Absolute positions, relative positions and a middle ground

The original transformer added fixed sinusoidal vectors to the token embeddings. That is simple and cheap, but it encodes absolute position: the signal for token 7 is the same whether token 7 sits in the middle of a short prompt or at the start of a long document. What attention actually needs is usually relative — how far apart two tokens are, and in which direction.

A second family of methods adds a learned or computed bias directly to the attention logits, as a function of the distance between query and key. That is distance-aware, but touching the score matrix costs compute and complicates fused attention kernels.

RoPE sits between the two. It modifies only the query and key vectors, before scores are computed, yet the scores that come out depend on relative distance. No bias term, no extra matrix, no change to the shape of anything downstream.

The core idea: rotate instead of add

Split each head’s query and key vector into consecutive pairs of dimensions and read each pair as a point in a 2D plane. For a token at position m, rotate every pair by an angle proportional to m. Later positions rotate further. The vector keeps its length; only its direction changes.

The payoff is in the dot product. When you take the inner product of a query rotated by angle m·θ and a key rotated by n·θ, the two rotations partially cancel. What remains is equivalent to the unrotated query compared against the key rotated by (n − m)·θ. This is just the cosine angle-difference identity applied pairwise. The absolute indices disappear from the result and only their difference survives.

That is the shift-invariance property in one line: slide the whole sentence 50 tokens to the right and the attention pattern between its tokens is unchanged, because every pair still has the same separation. The model learns distance rather than index numbers, exactly the generalisation you want.

One rotation speed per dimension pair

A single rotation frequency would be a poor encoder: it would wrap around quickly and confuse near and far tokens. Instead RoPE assigns each dimension pair its own frequency, decaying geometrically across the head dimension and controlled by a base parameter — 10,000 in the original formulation.

  • High-frequency pairs rotate quickly and resolve fine, local structure: adjacent words, short phrases, immediate syntax.
  • Low-frequency pairs rotate slowly and stay distinguishable over thousands of tokens, carrying long-range position information.

This multi-scale schedule is the reason one mechanism works for both a two-sentence prompt and a long document, and it is also the reason context extension is not a single global knob — different frequency bands need different treatment.

Where RoPE sits in the stack

Implementation is short. Precompute the inverse frequencies, broadcast them across position indices, take the cosine and sine, and combine them with a “rotate half” step that swaps and negates the paired dimensions. The rotated vector is the original scaled by the cosine term plus the swapped-and-negated version scaled by the sine term. Fifteen or so lines of tensor code.

Placement matters more than the code:

  • Apply RoPE after the Q, K and V projections and before the attention scores, per head.
  • Apply it to queries and keys only. Value vectors are not rotated; they carry content, not position. Rotating V is a quietly damaging bug.
  • With a KV cache, simply continue position ids from where the cache ended. Because keys are stored already rotated, cached entries stay valid and generation composes cleanly — one reason RoPE is comfortable in the inference stacks used to serve models on your own GPUs.
  • Some architectures use a partial rotary factor, rotating only a fraction of each head’s dimensions and leaving the rest untouched.
  • For images, the principle extends to two dimensions. In the axial form studied by Heo et al. (2024), half the head dimensions rotate with row position and half with column position; their RoPE-Mixed variant learns frequencies mixing both axes. Video needs a third, temporal axis; the Qwen2-VL paper (2024) introduced M-RoPE, splitting the rotary embedding into temporal, height and width components, and the Qwen3-VL technical report (2025) interleaves those axes across frequency bands.

Pushing past the training window

Naive extrapolation fails. Ask a model trained on, say, a 4k-token window to attend across eight times that length and the low-frequency pairs reach rotation angles the model has never encountered. Behaviour degrades sharply, because attention is computed on position signals never seen in training. Several principled approaches exist:

  • Position interpolation (Chen et al., 2023) compresses the new position indices back into the trained range, so angles stay in distribution. It works, but it packs neighbouring tokens closer together in angle space, which can blunt fine local discrimination.
  • NTK-aware scaling adjusts the base of the frequency grid instead of the positions, spreading the change unevenly across bands so the rotation schedule stays smooth.
  • YaRN (Peng et al., 2023) and LongRoPE (Ding et al., 2024) go further by treating frequency bands differently: scale the low-frequency, long-range dimensions aggressively while leaving the high-frequency, local dimensions close to their original behaviour.

All of these trade something. Extended context is not free capability, and a model stretched to a long window still has to be evaluated on the long-context tasks you actually care about, alongside the short prompts that make up most production traffic. The same tension drives architectural work such as hybrid attention in long-context models.

How Meta’s use of RoPE has changed

Meta’s own models show how the role of RoPE is shifting. LLaMA, Llama 2 and Llama 3 apply it in every attention layer. With Llama 4 (April 2025), Meta introduced what it calls the iRoPE architecture: rotary embeddings in most layers, interleaved attention layers with no positional embedding at all, and temperature scaling of attention at inference time.

The Muse family from Meta Superintelligence Labs goes a step further. Meta’s model card for the open-weight Muse Glimmer-30B (10 August 2026) lists RoPE with a base of 500,000 in local layers only; those layers use sliding-window attention in a repeating pattern of three local layers and one global layer, and the global layers carry no positional encoding. Muse Spark Open, whose weights Meta released on 15 October 2026, follows the same design: its sliding-window layers are rotated, and every fourth, full-context layer drops RoPE. In practice, rotary settings are now defined per layer type rather than once for the whole model.

What this means for teams building and running their own models

RoPE is a small piece of code with outsized consequences, so treat it as a configuration surface rather than a detail:

  • Record the rotary settings with the checkpoint. Base, partial rotary factor, any scaling method and, where the architecture has them, per-layer settings must match between training, fine-tuning and serving. A mismatch produces a model that looks loaded correctly and reasons badly.
  • Measure short-context quality after extending context. Regression here is the usual cost of aggressive scaling and is easy to miss if you only benchmark retrieval over long inputs.
  • Build evaluation sets at the lengths you will really see. If your documents are 60k tokens, a 4k benchmark tells you nothing about the configuration you are about to ship.
  • Check position bookkeeping across cached generation, batching and any prompt truncation logic, since off-by-one position ids degrade output subtly rather than crashing.

AMD Telecom trains and fine-tunes deep-learning models and large language models on its own in-house GPUs, and collects training data with BigParser, its in-house platform that turns web content and documents into structured training sets.

If you are planning work that depends on long inputs or on adapting an existing model to your own domain, our AI Model Training and AI Model Development services cover the path from dataset to deployed model. Contact us to discuss your use case.