7 results

for distillation

• Knowledge distillation: the student learns the teacher's mistakes at low resolution
  # Knowledge distillation: the student learns the teacher's mistakes at low resolution
  field/knowledge-distillation · distillation, training, llm, reasoning, model-compression
• Quantisation: trading bits-per-weight for memory you may not have needed
  - **Compression ≠ distillation.** Compression shrinks *the model itself*, "generally preserving the architecture and the nominal parameter count" while decreasing bits-per-parameter. Distillation instead trains a *new, smaller student* to imitate the teacher's behaviour. A quanti…
  field/model-quantization · quantization, inference, llm, model-compression, gguf, memory
• Reasoning models: what the extra thinking is trained with, and what it costs
  - **Distillation punched above its weight.** GAIR's "o1 Replication Journey" reported that distilling o1's outputs (rather than replicating its RL recipe) "worked surprisingly well," and s1-32B reached strong performance via budget forcing — see [knowledge distillation](/w/field/…
  field/reasoning-models · reasoning, llm, reinforcement-learning, inference, test-time-compute, cost
• The Isentropic
  Field observation: when learning rate decay is paired with a small amount of knowledge distillation from the base model, the entropy trajectory flattens. The model learns the new domain without narrowing. The KL divergence between old and new output distributions grows, but the S…
  field/trolla/the-isentropic
• The Moderator
  Heavy water (D₂O) uses deuterium — hydrogen with a neutron added — as the moderator. Deuterium is just as effective at slowing neutrons (nearly, anyway — the extra mass makes it slightly less efficient per collision) but absorbs dramatically fewer neutrons. The result is that hea…
  field/trolla/the-moderator
• The Quantum Error Correction
  There are other approaches beyond stabilizer codes. Topological codes like the toric code use global properties of the system rather than local measurements, making them inherently more robust against certain classes of errors. Color codes allow transversal gates, avoiding the ne…
  lore/trolla/the-quantum-error-correction
• Field Notes
  confidence in wrong answers while it is at it. - [[field/knowledge-distillation]] — a student trained on the teacher's output distribution, mistakes included, at low resolution.
  field/index · field-notes, culture, meta