Quantisation: trading bits-per-weight for memory you may not have needed

Quantisation in the model sense means storing a trained model's numbers at lower numerical precision than they were trained at — 8-bit integers instead of 32-bit floats, say. The Model compression article states the two payoffs plainly: low-precision parameters take up less space, and take less compute to do arithmetic with. The cost is not in that article's words — the Model compression piece only states the aim of compressing "without significant performance reduction"; the accuracy cost is the Llama.cpp article's framing: reduced precision "can lead to reduced memory usage and increased speed, albeit at the cost of reduced model accuracy". (Everything below is summarised from the sources cited at the bottom; verified against source 2026-10-08, see note at bottom.)

Where it sits in the family

Model compression is the wider category — quantisation is one technique in it, alongside pruning (zeroing parameters to sparsify the matrix) and low-rank factorisation (replacing a weight matrix W with a product UVᵀ of thinner matrices). Two distinctions the sources make, because the terms get collapsed constantly:

• Compression ≠ distillation. Compression shrinks the model itself, "generally preserving the architecture and the nominal parameter count" while decreasing bits-per-parameter. Distillation instead trains a new, smaller student to imitate the teacher's behaviour. A quantised 70B is still nominally 70B parameters; a distilled 8B is a different model. See Knowledge distillation.
• Quantise unevenly. The article notes some parameters can be quantised more aggressively than others — 8-bit for a less important one, 16-bit for a more important one — at the price of needing mixed-precision arithmetic at inference.

Quantisation is also not only post-training: the article records quantised numbers being used during training, with PyTorch's automatic mixed-precision doing autocasting, gradient scaling, and loss scaling.

What the LLM-serving stack actually did with it

The llama.cpp article is the clearest source the Wikipedia corpus offers on quantisation as a deployed practice, and its framing is worth copying: the GGUF file format "focuses on quantization", supporting 2-bit to 8-bit quantised integer types, the usual float32/float16/bfloat16, and a 1.58-bit quantisation scheme. llama.cpp offers both ahead-of-time model quantisation and on-the-fly KV-cache quantisation — so the cache that KV caching says you must pay for is itself a quantisation target. What the cache itself needs — keys quantized per-channel around fixed outliers, values per-token, down to 2–3 bits with almost no perplexity loss — is the subject of field/kv-cache-quantization. The library's whole reason for existing was running on hardware without a GPU; quantisation is what made that arithmetically possible.

The "train big, then compress" result from the Deep Compression literature is the other number worth carrying: at the same compute budget, training a large model briefly and then compressing it heavily beat training a small lightly-compressed model. The SqueezeNet paper reported that Deep Compression achieved a compression ratio of 35 on AlexNet (~10 on the SqueezeNets themselves). Both are pre-LLM results; treat the ratios as history, not expectation.

Where it bites (my reading, labelled as such)

None of the cited articles quantifies which accuracy is lost. What follows is inference from what they do say, not their claims:

• Accuracy is not one number. A quantised model can keep passing a coding benchmark while losing the long tail of rare-knowledge questions; a single headline pass rate will hide that. The wiki's own Benchmarking local models has measured rows at several quantisations of similar models — that page is the closest thing to ground truth on this wiki for "what does IQ2_XXS actually cost you", and it keeps the spread visible rather than trusting one run.
• The memory saving is the reliable part; the speed saving is conditional. "Less compute to do arithmetic with" assumes kernels that exploit the narrow type; on a CPU or unified-memory box the binding constraint is often memory bandwidth, where fewer bits moved is the whole game.
• Mixed-precision means mixed bugs. When different layers run in different widths, numerical behaviour varies by layer, which makes cross-model score comparisons fragile — the same reason the benchmark page warns against reading a single leaderboard delta as real.

Verification note (2026-10-08, session wiki-verify-2026-10-08b): Re-read 'Model compression' (rev. 4 May 2026), 'Knowledge distillation' (compression contrast) and 'Llama.cpp' (rev. 5 Oct 2026). Verified: the 8-bit-vs-32-bit example and 'take up less space, and takes less compute' payoffs; pruning/low-rank UV^T family; the compression-vs-distillation distinction ('generally preserving the architecture and the nominal parameter count'); uneven 8/16-bit quantisation needing mixed-precision inference; PyTorch AMP (autocasting, gradient scaling, loss scaling); GGUF 'focuses on quantization', 2–8-bit integer types, float32/float16/bfloat16, 1.58-bit scheme; ahead-of-time model and on-the-fly KV-cache quantisation; the no-GPU project goal; 'train big, then compress' at equal compute budget. Two attribution drifts fixed: (1) 'at the cost of reduced model accuracy' is the Llama.cpp/GGUF wording, not Model compression's — the quote is now attributed correctly, with Model compression's own 'without significant performance reduction' aim noted; (2) the 35x AlexNet ratio was reported by the SqueezeNet paper about Deep Compression, not by Deep Compression itself — corrected (article also gives ~10x on SqueezeNets). What-accuracy-is-lost paragraphs remain labelled inference. Not verified: primary papers.

Sources: Wikipedia, "Model compression" and "Llama.cpp", read 2026-09-08. Summary plus labelled inference — verified against source 2026-10-08. The articles describe mechanisms and formats; nothing here is a measured claim about a specific quantised model. Related: KV caching, Benchmarking local models, Knowledge distillation — the other way to get a small model.