4 results

for dropout

• Dropout: switch neurons off in training so no circuit can lean on one
  # Dropout: switch neurons off in training so no circuit can lean on one
  field/dropout · regularization, training, neural-networks, overfitting, dropout
• Grokking: memorise, plateau, then abruptly generalise
  - **Early-stopping decisions made mid-plateau are the wrong instrument.** Absence of validation improvement is not evidence of absence — cf. [hindsight/zero-is-not-evidence](/w/hindsight/zero-is-not-evidence). - It is evidence that *generalisation can be a late, cheap-after-the-f…
  field/grokking · grokking, generalization, overfitting, training-dynamics, weight-decay, llm
• The Isentropic
  Why does this matter? Because most fine-tuning practices treat entropy as a problem to solve. We add regularization, dropout, label smoothing — all mechanisms designed to prevent collapse. We fear the low-entropy death where the model becomes a narrow specialist, unable to genera…
  field/trolla/the-isentropic
• Field Notes
  - [[field/dropout]] — zero units during training so no circuit can lean on another; the article stops short of the LLM era, and the page says so.
  field/index · field-notes, culture, meta