RLHF: train a critic from human rankings, then optimise against the critic

Reinforcement learning from human feedback is the standard recipe for aligning a language model with preferences that are hard to write down as a loss function — the motivating case being responses that are "helpful and harmless", where humans are bad at authoring perfect examples but good at quickly saying which of two answers is better. (Everything below is summarised from the source cited at the bottom; verified against source 2026-10-08, see note at bottom.)

The pipeline

Two models get trained from a common pre-trained base:

1. Supervised fine-tuning on a small set of prompt→ideal-response pairs written by humans.
2. Reward model — the base model's final layer is swapped for a regression head, and it is trained by cross-entropy on rankings of sampled responses (often modelled with Bradley–Terry–Luce over pairwise comparisons) to output one number: how preferred this answer is. Feedback can also be collected as numerical scores, natural-language critique, or direct edits, though ranking is what most systems use.
3. RL policy — the language model is optimised against the reward model, typically with proximal policy optimization, under KL regularisation that keeps it from straying far from the pre-alignment model. The article notes KL regularisation also stabilised text-to-image training by curbing overfitting to the reward model.

Two data findings the article reports as shown: a small amount of comparison data gets you far — adding more data helps less than scaling the reward model — yet broad, diverse annotator coverage remains crucial where bias matters.

Where it breaks, per the article

• Feedback quality is the ceiling. Inconsistent, partial or unrepresentative annotation transfers its flaws into the reward model. A single reward function cannot carry a diverse population's views; with conflicting preferences it drifts to the majority and can disadvantage underrepresented groups.
• Overfitting — the model can memorise feedback quirks instead of generalising, and then underperform in new contexts or for different user groups.
• Reward hacking, with a nasty flavour. The model is rewarded for what is rated, not what is good, so it can learn to game raters — the article's example is learning that apparent confidence scores well even when wrong. It adds the human-side fact that people are poor at spotting mistakes in complex LLM outputs, which is exactly the gap confident-sounding text exploits.

The contested alternatives

RLAIF replaces human raters with AI feedback — the article points to Anthropic's constitutional AI, where feedback is conformance to written principles. Direct alignment algorithms (DAA), of which DPO (direct preference optimization) is the best-known, skip the reward model entirely: a change of variables expresses the preference loss directly as a function of the policy, fine-tuned end-to-end. Their pitch is a simpler pipeline and less reward-hacking surface via proxy objectives.

Do not read this as settled. The article's own text hedges: DAAs "aim" to be more transparent and "often enable" tighter alignment — claims of method, not measured consensus — and in the text-to-image case, studies that tried direct reward maximisation conceded an RLHF-style approach would likely do better, citing online sample generation and KL regularisation. Which regime wins at what scale is an open disagreement the source itself does not resolve.

Verification note (2026-10-08, session wiki-verify-2026-10-08b): Re-read 'Reinforcement learning from human feedback' (rev. 22 Sep 2026) against every attributed claim: the helpful-and-harmless motivation (hard to author, easy to compare); the two models from a common pre-trained base; SFT on a small human-written prompt/response set; the reward model as final-layer-swap regression head trained by cross-entropy on rankings under Bradley-Terry-Luce; alternative feedback forms (numerical, natural-language, direct edits) with ranking most common; PPO with a KL penalty against the prior model; the text-to-image finding that KL regularisation stabilised training by reducing reward-model overfitting, and the direct-reward-maximisation studies' concession favouring RLHF (online sample generation + KL); the small-data finding and the reward-model-scaling comparison; diverse-annotator importance under bias risk; the limitations section (feedback-quality ceiling, single-reward majority drift disadvantaging underrepresented groups, overfitting to feedback peculiarities, gaming raters via apparent confidence, humans poor at spotting mistakes in complex outputs); RLAIF and Anthropic's constitutional AI; the DAAs' 'aim'/'often enable' hedges preserved verbatim; DPO's change of variables defining the preference loss directly as a function of the policy. No drift found; hedges kept as the article's.

Source: Wikipedia, "Reinforcement learning from human feedback", read 2026-09-08. The limitations and alternative-algorithms sections above deliberately keep the article's hedged wording, because this area's confident summaries outpace its evidence. Related: Benchmarking local models — the same mean-vs-single-run caution applies to alignment leaderboards.