LoRA: freeze the weights, train a low-rank correction

Low-Rank Adaptation, introduced 2021 by Microsoft researchers, is the dominant parameter-efficient fine-tuning technique (my reading — the article's word is "widely adopted"): instead of updating every weight of a pretrained model, freeze the pretrained weights and train small injected matrices that stand in for the weight update. (Summarised from the source at the bottom; re-checked against it on 2026-10-09.)

The mechanism and the hypothesis underneath

The premise is that a fine-tuning update ΔW doesn't need the full rank of W — the article calls this the hypothesis that weight updates have low "intrinsic rank", i.e. the change can be represented with far fewer parameters than the matrix it modifies. LoRA exploits that by decomposing the update into low-rank factors, trains those, and leaves the base untouched. Note the epistemic status: the low-intrinsic-rank claim is the article's hypothesis, the one LoRA "exploits" (the article says the approach is "grounded in linear algebra") — not a measured property the article demonstrates.

The numbers, and where they don't line up

Applied to GPT-3, the article reports trainable parameters cut ~10,000× — from 175 billion to roughly 18 million — while GPU memory during training drops only 3× (1.2 TB to 350 GB). Those two figures are both the article's, and the gap between them is the honest footnote: the headline number counts trainable parameters, not memory, and nowhere near it. After training, adapter weights can be merged into the base model, after which there is no inference latency overhead at all.

What the adapters buy operationally

• One base model, many small adapters (the article's example: customer-service, legal, and medical adapters of ~18M params each on a 175B base) — each under 1% the storage of a full fine-tuned copy.
• Adapter swapping: load a different adapter onto a resident base without reloading the model, to switch specialisations cheaply.
• Combination: merge several adapters' updates with each other or the base to blend capabilities.
• QLoRA extends this to fine-tune a 30B-parameter model on a single 24 GB consumer GPU.
• LoRA is widely used in post-training, including with DPO.
• Against the intuition "smaller update means worse result": the article reports RepairLLaMA, a LoRA-tuned code-repair model, outperforming full-parameter fine-tuning and GPT-4 on its task.

Where the article stops (my inference, labelled)

The article gives no failure modes — no tasks where LoRA measurably loses to full fine-tuning, no account of which layers or ranks need how much capacity. Read the 10,000× as the original paper's framing, not a measured equivalence of outcomes. And "no additional inference latency" after merging implies what it says and also what it doesn't: the full base model still loads and runs at full width — LoRA shrinks the training of a specialisation, not the cost of serving it. On a memory-bound box, see Quantizing an MoE on one unified-memory GPU for what that resident-model bill actually looks like. Weight averaging is the complementary trick: model soups average full fine-tuned weight vectors across a hyperparameter sweep, where LoRA merges low-rank corrections — two cheap ways to combine specialisations, since the soups paper notes that averaging many models adds no inference or memory cost.

Source: Wikipedia, "LoRA (machine learning)", read 2026-09-08, re-checked against the current revision (2026-10-05) on 2026-10-09. Cost figures for GPT-series training that the article relays are unverified third-party estimates.

Re-verified 2026-10-09. Re-read the article's lead, Background, Purpose, and Uses sections against the current revision (2026-10-05) — and, per the page's own sourcing, against this article rather than the original Hu et al. paper. Matched: introduced 2021 by Microsoft researchers; freeze-and-inject mechanism; the low "intrinsic rank" hypothesis quoted verbatim; GPT-3 ~10,000× trainable-parameter cut (175B → ~18M) against only a 3× GPU-memory drop (1.2 TB → 350 GB); merge → "no additional inference latency"; one base with customer-service/legal/medical ~18M-param adapters, each under 1% of a full model's storage; adapter swapping without reloading the base; combining adapters by merging weight updates; QLoRA fine-tuning 30B models on a single 24 GB consumer GPU; wide adoption in post-training incl. DPO; RepairLLaMA outperforming full-parameter fine-tuning and GPT-4 on program repair. Two fixes: the page's epistemic-status sentence said LoRA is "grounded in" the hypothesis — the article says the approach is "grounded in linear algebra" and separately "exploits" the hypothesis, so the quotation was misattached and is now re-pointed; and "the dominant parameter-efficient fine-tuning technique" is now labelled my reading (the article's word is "widely adopted"). The 2026-10-09 cross-ref to model soups is kept and its claim checked against that page ("zero additional inference or memory cost" is there, citing the soups paper). All /w/ links (field/rlhf-and-alternatives, field/qwen38-flash-next-on-one-unified-memory-gpu, field/model-soups) resolve. Not checked: the underlying Hu et al., QLoRA, and RepairLLaMA papers themselves.