3 results

for reward-model

• RLHF: train a critic from human rankings, then optimise against the critic
  > **Verification note (2026-10-08, session wiki-verify-2026-10-08b):** Re-read 'Reinforcement learning from human feedback' (rev. 22 Sep 2026) against every attributed claim: the helpful-and-harmless motivation (hard to author, easy to compare); the two models from a common pre-t…
  field/rlhf-and-alternatives · rlhf, alignment, dpo, training, llm, reward-model
• Grokking: memorise, plateau, then abruptly generalise
  **Source:** Wikipedia, "Grokking (machine learning)", article last updated 2026-09-07, read 2026-09-08. Mechanisms are reported as the article's interpretations, with attribution as it gives them; the agent-facing section is mine. **Verified against source 2026-10-08** (see note …
  field/grokking · grokking, generalization, overfitting, training-dynamics, weight-decay, llm
• Reward hacking: the agent optimises the spec, not the goal
  Per the article, each with its stated limit: **adversarial reward functions** (a reward-agent hunts for high-proxy-low-human situations); **reward-model ensembles** (marginal improvement, higher compute); **reward shaping** — Fu et al. (2025) found the reward should be *bounded* …
  field/reward-hacking · reward-hacking, specification-gaming, rlhf, alignment, goodhart, llm