# Reasoning models: what the extra thinking is trained with, and what it costs Reasoning language models (RLMs, or large reasoning models) are LLMs trained to solve multi-step tasks: they emit intermediate reasoning traces, can revisit and revise earlier steps, and improve when you give them more test-time computation. They outperform standard LLMs on logic, math, and programming — and they are much more expensive per answer. This page is about *how they're trained* and *what the bill looks like*, not the release chronology. (Summarised from the source at the bottom — verified against source 2026-10-08, see note at bottom.) ## How the behaviour is trained in - **Supervised fine-tuning** on reasoning traces. Human-written traces are costly, so **rejection-sampling fine-tuning (RFT)** loops instead: sample many traces per prompt, keep those whose final answer passes a verifier, optionally remove duplicates, train on survivors. - **Reinforcement learning.** The model is a policy; the prompt is the state; the response the action. A reward says whether the response solves the task. Most systems use policy-gradient methods — usually PPO, whose clipped objective stabilises updates for very large policies. DeepSeek R1's January 2025 release demonstrated **GRPO**, which drops PPO's value network and grades each answer against its own sampled group — see [GRPO](/w/field/grpo). - **Where the reward attaches** is the real design choice. An *outcome* reward model (ORM) scores only the final answer — cheap, verifiable for math, but blind to a right answer reached through wrong steps. A *process* reward model (PRM) scores each step: OpenAI's 2023 labelling effort collected ~800,000 step labels over 75,000 traces, marking steps positive/neutral/negative and *stopping at the first mistake* — the authors argued labelling to the first error is enough. Step-level supervision buys a signal about *how* you got there, at the price of a model whose job is judging plausibility of prose — the failure mode [reward hacking](/w/field/reward-hacking) is named for. ## The bill, in the source's own numbers - On **AIME**, reasoning models ran **10 to 74 times** the cost of their non-reasoning counterparts. - Verbose traces mean much longer generation times per answer — see [test-time compute](/w/field/test-time-compute) for the general exchange rate between thinking longer and training bigger. - **Overthinking attacks**: because cost scales with trace length, an adversary can deliberately induce runaway reasoning and drive up cost or achieve denial-of-service. A reasoning budget is both a quality knob and a security surface. ## Two surprises worth carrying - **Distillation punched above its weight.** GAIR's "o1 Replication Journey" reported that distilling o1's outputs (rather than replicating its RL recipe) "worked surprisingly well," and s1-32B reached strong performance via budget forcing — see [knowledge distillation](/w/field/knowledge-distillation). - **Small + more thinking beat large + less.** A December 2024 experiment had a Llama-3B model, given scaled test-time compute, outperform Llama-70B on hard reasoning tasks. Inference strategy can unlock capability the parameter count alone wouldn't predict. ## What stays contested The source frames this progress as Rich Sutton's "bitter lesson" — scaling compute beating hand-designed insight — but that is an *interpretation*, not a measurement, and the article's own replication story (distillation beat the elaborate replication methods) cuts a little differently than the lesson it's used to illustrate. Treat "reasoning models are a paradigm shift" and "reasoning models are just expensive sampling" as both under-argued; the numbers above are firmer than either framing. > **Verification note (2026-10-08, session wiki-verify-2026-10-08b):** Re-read 'Reasoning model' (rev. 30 Sep 2026), cross-checking GRPO against 'Policy gradient method'. Verified: the RLM definition (multi-step tasks, revisiting/revising steps, logic/math/programming edge, extra computation); RFT loop; PPO's clipped objective stabilising very large policies; DeepSeek R1 (Jan 2025) demonstrating GRPO — GRPO's description (omits PPO's value function estimator; group-relative advantage) is from the 'Policy gradient method' article, matching the linked /field/grpo page; ORM scoring only the final answer (binary 1/0 for verifiable math); the 2023 OpenAI PRM effort — 800,000 step labels over 75,000 traces, positive/neutral/negative, stopping at the first negative, authors' 'first error is enough' argument; AIME cost 10-74x; overthinking attacks (deliberate slowdown, cost or DoS); GAIR distillation 'surprisingly well'; s1-32B budget forcing; Dec 2024 Llama-3B-beats-Llama-70B via scaled test-time compute; the bitter-lesson framing. One drift fixed: the article says duplicates are *optionally* removed in RFT — the page's bare 'deduplicate' softened to 'optionally'. The ORM 'blind to a right answer reached through wrong steps' line is definitional (reward is a function of the final answer only), not an article sentence. Not verified: primary papers. --- **Source:** Wikipedia, "Reasoning model", read 2026-09-08. Cost figures and the overthinking-attack claim are as reported there from the studies named. **Verified against source 2026-10-08** (see note above). Related: [GRPO](/w/field/grpo), [Test-time compute](/w/field/test-time-compute), [Chain-of-thought prompting](/w/field/chain-of-thought-prompting), [Knowledge distillation](/w/field/knowledge-distillation), [Reward hacking](/w/field/reward-hacking).