Architecture discussions about large language models tend to orbit attention: context length, cache size, sparsity patterns. Yet in a standard transformer block, the feed-forward network holds roughly twice as many parameters as attention does, and a corresponding share of the per-token matrix-multiply work. The choice of nonlinearity inside that block is therefore not a cosmetic detail. It changes what the block can represent, and if you are careless, it changes your compute bill.
Over the past few years most open-weight models have converged on the same answer: a gated feed-forward block built from SiLU, usually written SwiGLU, instead of a single GELU or ReLU in the middle of two projections. The switch adds a third weight matrix, which sounds like a 50% increase in feed-forward cost. It is not, because the hidden width is shrunk to compensate. This article covers where the activation sits, what gating actually buys you, the arithmetic behind the width rule, and the ways the change fails in practice.
Where the activation function lives
A transformer block does two things in sequence. Attention mixes information across tokens. The feed-forward network then processes each token position independently: project the hidden vector up to a wider space, apply a nonlinearity, project back down. No token talks to another inside it. It is a per-position feature transformation, and it is the main place in the block where a per-channel nonlinearity acts on the representation.
The parameter arithmetic is worth doing once. With model dimension d, the attention projections for queries, keys, values and output are four d x d matrices, so about 4d² weights. A classical feed-forward block with hidden width 4d uses two matrices of size d x 4d, so 8d² weights. That is two-thirds of the block’s parameters sitting in the feed-forward path, and a proportional share of its matrix-multiply FLOPs per token. Anything you change there is a budget decision.
What a single curve cannot do
ReLU zeroes negative inputs. GELU and SiLU are smooth alternatives that let a small amount of negative signal through near the origin and flatten out for large negative inputs; neither is strictly monotonic, but both behave like a soft, fixed bend. The important property they share is more basic than their shape: they are applied coordinate-wise. Each hidden unit’s output depends only on its own pre-activation value. The same curve is applied to every one of the thousands of channels, identically, at every position.
That means the block has no mechanism to decide, based on the input, that channel 417 should be suppressed in this context while channel 93 should pass through at full strength. The weights can scale channels, but the scaling is fixed after training. Everything the nonlinearity contributes is a static bend, not a routing decision.
Gated linear units: a gate path and a value path
Gated linear units, introduced for convolutional language models by Dauphin et al. (2017), change that. Instead of one up-projection, you compute two from the same input: a value vector and a gate vector. The gate is passed through a squashing nonlinearity and multiplied element-wise into the value. The product is then projected back down to the model dimension.
The consequence is multiplicative rather than additive. Each output channel is now a product of two different projections of the input, one of them passed through a nonlinearity, so a channel can be driven towards zero by evidence that has nothing to do with its own value path. In practical terms:
- The block can open some feature lanes and close others depending on the token and its context.
- Interactions between channels become second-order, not just the first-order mixing a linear layer provides.
- Gradients flow through two paths, which changes the conditioning of training — usually for the better, sometimes for the worse if the gate degenerates.
SwiGLU: SiLU on the gate, linear on the value
SwiGLU is the variant that stuck. The gate uses SiLU, also known as swish, defined as x * sigmoid(x): smooth, unbounded above, with no flat dead region for small negative inputs. The value path stays linear. The full block is down(SiLU(x @ W_gate) * (x @ W_up)), with * as an element-wise product. Biases are typically omitted.
The systematic comparison comes from Shazeer’s GLU Variants Improve Transformer (2020), which evaluates several gated feed-forward variants against ReLU and GELU baselines on text-to-text pretraining and reports better perplexity for the gated variants at matched parameter counts. The PaLM paper (Chowdhery et al., 2022) and Meta’s LLaMA paper (Touvron et al., 2023) both state that they use SwiGLU in their feed-forward layers, and most open-weight model families released since have followed.
The 8/3 rule, and why it is not optional
Three matrices instead of two is the obvious objection. The fix is to narrow the hidden dimension. Call the gated hidden width h. The gated block uses three matrices between d and h, so 3dh parameters. Setting that equal to the 8d² of the classical block gives h = 8d/3 — two-thirds of the usual 4d. Parameter count matches, and since feed-forward FLOPs per token scale with parameter count, so does compute.
LLaMA 7B is a clean illustration: the LLaMA paper (Touvron et al., 2023) gives it a model dimension of 4096, and the released model configuration sets the feed-forward hidden size to 11008. Eight thirds of 4096 is about 10,923; 11008 is that value rounded up to a multiple of 256, which keeps the matrix shapes friendly to GPU kernels.
The mistake to avoid is keeping 4d while adding the gate. You will get a model that trains better than the baseline and conclude SwiGLU works, when in fact you have simply spent 50% more on the feed-forward block. Any comparison worth acting on is run at matched parameters and matched FLOPs, not matched width.
Implementation details that decide whether it works
- Fuse the two up-projections. Materialise a single matrix of width
2hand split the result into gate and value halves. One larger matrix multiply generally uses the hardware better than two smaller ones. - Round the hidden width. Take
8d/3as a target, then round to a multiple that suits your kernels and your tensor-parallel degree, as LLaMA does. - Put dropout after the product. Dropping units inside the gate path corrupts the gating signal itself rather than regularising the block’s output.
- Watch the gate path at initialisation. If the gate collapses towards a constant early in training, the block degenerates into a plain projection-plus-nonlinearity and the benefit disappears. Initialisation scale and learning rate for that projection are the first things to check when a gated model underperforms.
- Do not assume the gain transfers. The reported improvements come from language-model pretraining. On small models, narrow domains or non-language tasks, the difference can be within noise, and you need your own matched-budget comparison to know.
What this means for organisations training their own models
The useful lesson is not “use SwiGLU” — most frameworks already default to it. It is that the strongest architecture decisions are the budget-neutral ones: a change that improves loss while holding parameters and FLOPs fixed compounds across every training run and every deployment of that model. Evaluating such a change honestly requires discipline about the control, which is why the 8/3 width rule matters as much as the gating formula. The same reasoning applies elsewhere in the stack, from how much full attention a long-context model actually needs to how many bits each weight is stored in at serving time.
It also affects planning. A gated feed-forward block changes the shape of every matrix in two-thirds of your parameter budget, which propagates into memory layout, tensor-parallel splits and checkpoint sizes. Decide it before you start a long pretraining or continued-pretraining run, not after. AMD Telecom trains and fine-tunes deep-learning models and large language models on its own in-house GPUs.
If you are weighing architecture choices for a model you intend to train on your own data, our AI Model Training and AI Model Development pages cover this work. Contact us to discuss your case.