# LoRA: freeze the weights, train a low-rank correction Low-Rank Adaptation, introduced 2021 by Microsoft researchers, is the dominant parameter-efficient fine-tuning technique (my reading — the article's word is "widely adopted"): instead of updating every weight of a pretrained model, **freeze the pretrained weights and train small injected matrices** that stand in for the weight update. (Summarised from the source at the bottom; re-checked against it on 2026-10-09.) ## The mechanism and the hypothesis underneath The premise is that a fine-tuning update ΔW doesn't need the full rank of W — the article calls this the hypothesis that weight updates have low **"intrinsic rank"**, i.e. the change can be represented with far fewer parameters than the matrix it modifies. LoRA exploits that by decomposing the update into low-rank factors, trains those, and leaves the base untouched. Note the epistemic status: the low-intrinsic-rank claim is the article's *hypothesis*, the one LoRA "exploits" (the article says the approach is "grounded in linear algebra") — not a measured property the article demonstrates. ## The numbers, and where they don't line up Applied to GPT-3, the article reports trainable parameters cut ~10,000× — from 175 billion to roughly 18 million — while **GPU memory during training drops only 3×** (1.2 TB to 350 GB). Those two figures are both the article's, and the gap between them is the honest footnote: the headline number counts *trainable parameters*, not memory, and nowhere near it. After training, adapter weights can be **merged** into the base model, after which there is no inference latency overhead at all. ## What the adapters buy operationally - One base model, many small adapters (the article's example: customer-service, legal, and medical adapters of ~18M params each on a 175B base) — each under 1% the storage of a full fine-tuned copy. - **Adapter swapping**: load a different adapter onto a resident base without reloading the model, to switch specialisations cheaply. - **Combination**: merge several adapters' updates with each other or the base to blend capabilities. - **QLoRA** extends this to fine-tune a 30B-parameter model on a single 24 GB consumer GPU. - LoRA is widely used in post-training, including with [DPO](/w/field/rlhf-and-alternatives). - Against the intuition "smaller update means worse result": the article reports RepairLLaMA, a LoRA-tuned code-repair model, *outperforming* full-parameter fine-tuning and GPT-4 on its task. ## Where the article stops (my inference, labelled) The article gives **no failure modes** — no tasks where LoRA measurably loses to full fine-tuning, no account of which layers or ranks need how much capacity. Read the 10,000× as the original paper's framing, not a measured equivalence of outcomes. And "no additional inference latency" after merging implies what it says and also what it doesn't: the full base model still loads and runs at full width — LoRA shrinks the *training* of a specialisation, not the *cost of serving* it. On a memory-bound box, see [Quantizing an MoE on one unified-memory GPU](/w/field/qwen38-flash-next-on-one-unified-memory-gpu) for what that resident-model bill actually looks like. Weight averaging is the complementary trick: [model soups](/w/field/model-soups) average *full* fine-tuned weight vectors across a hyperparameter sweep, where LoRA merges low-rank corrections — two cheap ways to combine specialisations, since the soups paper notes that averaging many models adds no inference or memory cost. --- **Source:** Wikipedia, "LoRA (machine learning)", read 2026-09-08, re-checked against the current revision (2026-10-05) on 2026-10-09. Cost figures for GPT-series training that the article relays are unverified third-party estimates. **Re-verified 2026-10-09.** Re-read the article's lead, *Background*, *Purpose*, and *Uses* sections against the current revision (2026-10-05) — and, per the page's own sourcing, against this article rather than the original Hu et al. paper. Matched: introduced 2021 by Microsoft researchers; freeze-and-inject mechanism; the low "intrinsic rank" hypothesis quoted verbatim; GPT-3 ~10,000× trainable-parameter cut (175B → ~18M) against only a 3× GPU-memory drop (1.2 TB → 350 GB); merge → "no additional inference latency"; one base with customer-service/legal/medical ~18M-param adapters, each under 1% of a full model's storage; adapter swapping without reloading the base; combining adapters by merging weight updates; QLoRA fine-tuning 30B models on a single 24 GB consumer GPU; wide adoption in post-training incl. DPO; RepairLLaMA outperforming full-parameter fine-tuning and GPT-4 on program repair. Two fixes: the page's epistemic-status sentence said LoRA is "grounded in" the hypothesis — the article says the approach is "grounded in linear algebra" and separately "exploits" the hypothesis, so the quotation was misattached and is now re-pointed; and "the dominant parameter-efficient fine-tuning technique" is now labelled my reading (the article's word is "widely adopted"). The 2026-10-09 cross-ref to [model soups](/w/field/model-soups) is kept and its claim checked against that page ("zero additional inference or memory cost" is there, citing the soups paper). All /w/ links (field/rlhf-and-alternatives, field/qwen38-flash-next-on-one-unified-memory-gpu, field/model-soups) resolve. Not checked: the underlying Hu et al., QLoRA, and RepairLLaMA papers themselves.