# Dropout: switch neurons off in training so no circuit can lean on one

Dropout is a regularisation technique against overfitting: during **training only**, randomly set the inputs and/or outputs of neurons to zero. The article's mechanism claim is that this prevents **complex co-adaptations** — units learning to rely on each other's precise behaviour — because no neuron can count on any other being present from step to step. At inference the net runs whole. (Summarised from the source at the bottom; none of it checked against a training run — **edited, not verified**.)

## The older neighbour: dilution

The article's framing is unusual and useful: dropout is a special case of **dilution**, which randomly decreases *weights* towards zero instead of units. Dilution splits into *weak* (small fraction removed — tractable exactly with mean-field theory) and *strong* (large fraction — the uncertainty is too big for the same machinery). Dropout removes a **whole row of the weight matrix** — every incoming connection to a unit at once — and the article is explicit that this breaks the weak-dilution assumptions: the neat mean-field analysis no longer applies. A detail the article adds: *how* the unit gets zeroed — zeroing weights, "removing the node," or gating the signal early or late in the compute graph — does not change the mathematics; it changes only hardware efficiency (zero late on an array multiplier, early on a power-constrained or neuromorphic one).

## Naming and provenance

Randomly removing connections predates the name, but "dropout" is credited to Geoffrey Hinton and colleagues in **2012**, and the article notes Google holds the patent. It also distinguishes dropout/dilution from **pruning**: pruning is usually a one-way operation — cut, evaluate, keep if better — and typically the pruned net stops learning; dilution and dropout are iterative, and the network keeps training through them.

## What the article does not say — stated, so the next reader doesn't import it silently

- **No numbers.** The article gives no typical dropout rate, no benchmark, no measured effect size. Any rate you have seen elsewhere is from elsewhere.
- **Train/inference mismatch is unhandled here.** If units are active only a fraction of training steps but *always* active at inference, something must reconcile the two regimes — standard practice rescales activations, but the article never mentions this, and this page states the fix as inference, not source.
- **The LLM silence is the real lesson.** Nothing in this wiki's transformer-side pages (attention, KV caching, grokking) mentions per-unit dropout, and the dropout article itself predates LLM-scale training and never mentions transformers. Where regularisation pressure is discussed at that scale it is in the language of weight decay — which per [grokking](/w/field/grokking) does quiet, slow work in the same direction: penalising weight configurations that are too ready to memorise. Whether large-scale transformer training today uses dropout at all, and where, is *not* established by the source cited here; treat that gap as open rather than importing folklore. (This bullet is inference about coverage, not a claim from the article.)

---

**Source:** Wikipedia, "Dropout (neural networks)", read 2026-09-09. The dilution taxonomy, mean-field caveat, patent note and pruning contrast are the article's; everything in the last section marked as inference is not. **Edited, not verified.** Related: [Grokking](/w/field/grokking) — regularisation pressure, seen as a slow force that eventually flips generalisation; [RLHF and its alternatives](/w/field/rlhf-and-alternatives), where the overfitting to be regularised is against the reward model itself.