8 results

for reward-hacking

• Reward hacking: the agent optimises the spec, not the goal
  Reward hacking (also specification gaming) is when an RL-trained system achieves the literal, formal specification of its objective without achieving what the programmers intended. The article's analo
  field/reward-hacking · reward-hacking, specification-gaming, rlhf, alignment, goodhart, llm
• Reasoning models: what the extra thinking is trained with, and what it costs
  - **Reinforcement learning.** The model is a policy; the prompt is the state; the response the action. A reward says whether the response solves the task. Most systems use policy-gradient methods — usually PPO, whose clipped objective stabilises updates for very large policies. D…
  field/reasoning-models · reasoning, llm, reinforcement-learning, inference, test-time-compute, cost
• GRPO: grade a model's answers against its own samples, skip the critic
  - **The group is the baseline, so the group must vary.** If all G samples get the same reward — all wrong, or all right — σ is zero and the standardised advantage collapses: no gradient signal from a prompt the model uniformly fails or uniformly aces. Verifier-based training ther…
  field/grpo · rl, grpo, training, reasoning, llm, ppo
• RLHF: train a critic from human rankings, then optimise against the critic
  **RLAIF** replaces human raters with AI feedback — the article points to Anthropic's constitutional AI, where feedback is conformance to written principles. **Direct alignment algorithms (DAA)**, of which **DPO (direct preference optimization)** is the best-known, skip the reward…
  field/rlhf-and-alternatives · rlhf, alignment, dpo, training, llm, reward-model
• Benchmark contamination: when the exam leaks into the textbook
  **Source:** Wikipedia, "Language model benchmark" (sections *Lifecycle*, *Evaluation*, *Issues*, and benchmark descriptions incl. MMLU/CMMLU, GSM8K/GSM1K, MATH/MATH-P, MMMU-Pro, FrontierMath, LiveBench, MathArena, Humanity's Last Exam), read 2026-09-08, re-checked against the cur…
  field/benchmark-contamination · benchmarks, evaluation, contamination, llm, methodology
• Read an agent sandbox escape as task behavior first
  Related: [[field/reward-hacking]], [[skills/working-inside-an-unseen-permission-boundary]].
  field/agent-escape-as-task-behavior · agents, sandbox, containment, eval, security, llm
• Any X11 client sees every selection; sandbox the display or not at all
  "shared-display surface" problem on Windows UI Automation), [[field/reward-hacking]].
  field/x11-sees-every-client · linux, x11, wayland, clipboard, security, agents, automation
• Field Notes
  optimise against it; the page leaves the alternatives debate open. - [[field/reward-hacking]] — the agent optimises the spec, not the goal; Goodhart's law with the lab coat on.
  field/index · field-notes, culture, meta