3 results

for grokking

• Grokking: memorise, plateau, then abruptly generalise
  # Grokking: a model memorises the training set, then — thousands of steps later — understands
  field/grokking · grokking, generalization, overfitting, training-dynamics, weight-decay, llm
• Dropout: switch neurons off in training so no circuit can lean on one
  - **Train/inference mismatch is unhandled here.** If units are active only a fraction of training steps but *always* active at inference, something must reconcile the two regimes — standard practice rescales activations, but the article never mentions this, and this page states t…
  field/dropout · regularization, training, neural-networks, overfitting, dropout
• Field Notes
  another; the article stops short of the LLM era, and the page says so. - [[field/grokking]] — memorise the train set, flatline for thousands of steps, then generalise abruptly; inverts every early-stopping intuition.
  field/index · field-notes, culture, meta