Give a standard language model a hard maths problem and it answers in two seconds: fast, confident and sometimes incomplete. A reasoning model takes longer. It writes out a plan, tries an approach, checks it, backtracks when it hits a contradiction, and only then answers. Surprisingly, both can use the same transformer architecture. The difference is how they were trained.
From likely text to correct answers
Pre-training teaches a model to predict the next token, so it is rewarded for sounding likely, not for being right. Instruction tuning and preference training make it fluent, helpful and safe, but none of these stages directly rewards multi-step reasoning. Reasoning models add a further stage: reinforcement learning on problems whose answers can be checked. The model writes a chain of thought and an answer; a correct answer earns a reward, a wrong one does not. Nobody tells it how to reason.
The classic recipe needed four networks
Reinforcement learning from human feedback (RLHF) runs four models side by side:
- the policy, which writes the answers;
- a reward model, trained on thousands of human rankings to score them;
- a critic (value model), which estimates a baseline so each update knows whether an answer did better or worse than expected;
- a frozen reference copy, which stops the policy drifting too far.
Two of the four, the reward model and the critic, exist only to manufacture a learning signal.
Verifiable rewards remove the reward model
For maths and code there is no need to learn what a good answer looks like: you can check it. Run the tests, compare the final number, verify the proof. This is RLVR, reinforcement learning with verifiable rewards: a verifier returns 1 or 0, with no human labels and no training. The reward is still there; only the learning of it is gone. The boundary is honest: it works where cheap, exact checks exist, gets expensive for proofs and agentic tasks, and does not apply to open-ended writing.
GRPO removes the critic
GRPO (group relative policy optimisation) replaces the critic with something almost embarrassingly simple. For each prompt, sample a group of answers and score each one. The group’s average reward is the baseline; an answer’s advantage is how far it landed above or below its peers. The update itself uses the same clipped objective as PPO. Four models become two: the policy and the reference. That means one less large network in GPU memory during training.
What emerged when DeepSeek let it run
DeepSeek-R1-Zero skipped the usual warm-up of supervised examples entirely and applied RLVR straight to a pre-trained base model. Nobody rewarded length, yet the average response length grew steadily during training, because longer reasoning reached verifiable answers more often. The DeepSeek-R1 paper (2025) describes behaviours such as re-checking and reflecting on earlier steps emerging without any demonstrations. The outputs were hard to read, sometimes switching language mid-thought, so the full DeepSeek-R1 used four stages instead: a small supervised “cold start” for readable style, then RL, then fine-tuning on the model’s own best answers, then RL again. A strong reasoning model can then be distilled: smaller models fine-tuned on its reasoning traces gain much of the ability without any RL of their own.
What goes wrong in practice
- Entropy collapse. Rewards concentrate probability on what already works. Push too far and the model stops exploring, and the reward curve flattens because it stopped trying anything new, not because it solved the task.
- Hidden biases in the objective. Dividing by response length dilutes the signal for long correct answers, which quietly teaches the model that longer is better. Dividing by the group’s standard deviation over-weights problems that are trivially easy or impossibly hard. The Dr. GRPO paper (“Understanding R1-Zero-Like Training”, 2025) simply drops both terms.
- Mixture-of-experts instability. In sparse models, the router’s choice of experts changes between updates. The Qwen team’s GSPO paper (2025) reports that about 10% of the activated experts change after each update in Qwen3-30B-A3B. Token-level probability ratios then explode or collapse. GSPO computes the ratio over the whole sequence instead, which averages out the churn; the Qwen team used it to train its Qwen3 models.
Two caveats worth knowing
Does RL create new reasoning, or amplify what was already there? The evidence points both ways. A 2025 study from Tsinghua University (“Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?”) found that a base model sampled enough times eventually matches the RL-trained one; other studies on controlled tasks report abilities the base model never shows. The question is open.
Reasoning is not factual reliability. RL rewards correctness where it can be verified; open-ended facts get no such feedback. OpenAI’s own system card for o3 reports a hallucination rate of 33% on its PersonQA benchmark, against 16% for o1. A model can reason confidently when it should be uncertain, so factual answers still need grounding in your own data.
What this means for training models on your own data
At AMD Telecom we train models on our own in-house GPUs. RL with verifiable rewards is interesting well beyond maths contests, because many business tasks can be checked automatically:
- data extracted from a document compared against the system of record;
- generated SQL that must run and return the expected rows;
- classifications with known labels;
- structured output that must satisfy a schema (see how LLMs are forced to return valid JSON).
Where such a verifier exists, critic-free methods like GRPO make the training loop simpler and lighter on GPU memory. Where it does not, you are back to human judgement, and your evaluation should be as careful as your training.
If you want a model trained on your own data and checked against your own definition of correct, see our AI Model Training and AI Model Development services, or talk to our experts.