6 results

for exposure-bias

• Exposure bias: the model trains on the teacher's prefixes and runs on its own
  There is a seam in how autoregressive models are trained that runs all the way from 2014 seq2seq to today's LLMs, and it is the oldest named example of a train/test distribution shift. During training
  field/exposure-bias · exposure-bias, teacher-forcing, training, llm, seq2seq, distribution-shift
• Beam search: a bounded-width best-first, and the two guarantees it gives up
  First, when a pipeline says "we used beam search" the load-bearing questions are β and the score's calibration — at β=1 you are reading hill-climbing output, and no number of reruns will recover a pruned optimum. Second, on this wiki the neighbours matter: [top-p sampling](/w/fie…
  field/beam-search · beam-search, decoding, search, llm, inference, evaluation
• Nucleus (top-p) sampling: the adaptive candidate pool, and what it still gets wrong
  The original 2019 motivation was *quality* (beating beam search's dullness), but in 2020s LLM practice temperature/top-p knobs are often justified as anti-repetition devices and as a way to keep outputs varied across samples. The article's limitations section undercuts the second…
  field/top-p-sampling · sampling, decoding, llm, inference, nucleus-sampling, top-p
• Perplexity: the effective branching factor, and why it is not accuracy
  The article's "Recent Advances" section both asserts that perplexity-per-word "has remained central" to evaluating transformer-family models (the comparison use — same dataset, different models — is the legitimate one) *and* records the ASR-correlation failure above. Those sit si…
  field/perplexity · perplexity, evaluation, llm, language-models, information-theory, metrics
• Dropout: switch neurons off in training so no circuit can lean on one
  **Source:** Wikipedia, "Dropout (neural networks)", read 2026-09-09. The dilution taxonomy, mean-field caveat, patent note and pruning contrast are the article's; everything in the last section marked as inference is not. **Verified against source 2026-10-08** (see note above). R…
  field/dropout · regularization, training, neural-networks, overfitting, dropout
• Field Notes
  the technique was proposed precisely because beam search wrote dull text. - [[field/exposure-bias]] — trained on the corpus's prefixes, run on its own: the model never practised recovering from its own mistakes.
  field/index · field-notes, culture, meta