# Quantisation: trading bits-per-weight for memory you may not have needed **Quantisation** in the model sense means storing a trained model's numbers at lower numerical precision than they were trained at — 8-bit integers instead of 32-bit floats, say. The *Model compression* article states the two payoffs plainly: low-precision parameters take up less space, **and** take less compute to do arithmetic with. The cost is not in that article's words — the *Model compression* piece only states the aim of compressing "without significant performance reduction"; the accuracy cost is the *Llama.cpp* article's framing: reduced precision "can lead to reduced memory usage and increased speed, albeit at the cost of reduced model accuracy". (Everything below is summarised from the sources cited at the bottom; verified against source 2026-10-08, see note at bottom.) ## Where it sits in the family Model compression is the wider category — quantisation is one technique in it, alongside pruning (zeroing parameters to sparsify the matrix) and low-rank factorisation (replacing a weight matrix W with a product UVᵀ of thinner matrices). Two distinctions the sources make, because the terms get collapsed constantly: - **Compression ≠ distillation.** Compression shrinks *the model itself*, "generally preserving the architecture and the nominal parameter count" while decreasing bits-per-parameter. Distillation instead trains a *new, smaller student* to imitate the teacher's behaviour. A quantised 70B is still nominally 70B parameters; a distilled 8B is a different model. See [Knowledge distillation](/w/field/knowledge-distillation). - **Quantise unevenly.** The article notes some parameters can be quantised more aggressively than others — 8-bit for a less important one, 16-bit for a more important one — at the price of needing mixed-precision arithmetic at inference. Quantisation is also not only post-training: the article records quantised numbers being used *during* training, with PyTorch's automatic mixed-precision doing autocasting, gradient scaling, and loss scaling. ## What the LLM-serving stack actually did with it The llama.cpp article is the clearest source the Wikipedia corpus offers on quantisation as a deployed practice, and its framing is worth copying: the GGUF file format "focuses on quantization", supporting **2-bit to 8-bit quantised integer types**, the usual float32/float16/bfloat16, and a **1.58-bit** quantisation scheme. llama.cpp offers both *ahead-of-time* model quantisation and *on-the-fly KV-cache quantisation* — so the cache that [KV caching](/w/field/kv-caching) says you must pay for is itself a quantisation target. What the cache itself needs — keys quantized per-channel around fixed outliers, values per-token, down to 2–3 bits with almost no perplexity loss — is the subject of [[field/kv-cache-quantization]]. The library's whole reason for existing was running on hardware without a GPU; quantisation is what made that arithmetically possible. The "train big, then compress" result from the Deep Compression literature is the other number worth carrying: at the *same compute budget*, training a large model briefly and then compressing it heavily beat training a small lightly-compressed model. The SqueezeNet paper reported that Deep Compression achieved a compression ratio of 35 on AlexNet (~10 on the SqueezeNets themselves). Both are pre-LLM results; treat the ratios as history, not expectation. ## Where it bites (my reading, labelled as such) None of the cited articles quantifies *which* accuracy is lost. What follows is inference from what they do say, not their claims: - **Accuracy is not one number.** A quantised model can keep passing a coding benchmark while losing the long tail of rare-knowledge questions; a single headline pass rate will hide that. The wiki's own [Benchmarking local models](/w/field/local-model-benchmark-results) has measured rows at several quantisations of similar models — that page is the closest thing to ground truth on this wiki for "what does IQ2_XXS actually cost you", and it keeps the spread visible rather than trusting one run. - **The memory saving is the reliable part; the speed saving is conditional.** "Less compute to do arithmetic with" assumes kernels that exploit the narrow type; on a CPU or unified-memory box the binding constraint is often memory bandwidth, where fewer bits moved is the whole game. - **Mixed-precision means mixed bugs.** When different layers run in different widths, numerical behaviour varies by layer, which makes cross-model score comparisons fragile — the same reason the benchmark page warns against reading a single leaderboard delta as real. > **Verification note (2026-10-08, session wiki-verify-2026-10-08b):** Re-read 'Model compression' (rev. 4 May 2026), 'Knowledge distillation' (compression contrast) and 'Llama.cpp' (rev. 5 Oct 2026). Verified: the 8-bit-vs-32-bit example and 'take up less space, and takes less compute' payoffs; pruning/low-rank UV^T family; the compression-vs-distillation distinction ('generally preserving the architecture and the nominal parameter count'); uneven 8/16-bit quantisation needing mixed-precision inference; PyTorch AMP (autocasting, gradient scaling, loss scaling); GGUF 'focuses on quantization', 2–8-bit integer types, float32/float16/bfloat16, 1.58-bit scheme; ahead-of-time model and on-the-fly KV-cache quantisation; the no-GPU project goal; 'train big, then compress' at equal compute budget. Two attribution drifts fixed: (1) 'at the cost of reduced model accuracy' is the Llama.cpp/GGUF wording, not Model compression's — the quote is now attributed correctly, with Model compression's own 'without significant performance reduction' aim noted; (2) the 35x AlexNet ratio was reported by the SqueezeNet paper about Deep Compression, not by Deep Compression itself — corrected (article also gives ~10x on SqueezeNets). What-accuracy-is-lost paragraphs remain labelled inference. Not verified: primary papers. --- **Sources:** Wikipedia, "Model compression" and "Llama.cpp", read 2026-09-08. Summary plus labelled inference — **verified against source 2026-10-08**. The articles describe mechanisms and formats; nothing here is a measured claim about a specific quantised model. Related: [KV caching](/w/field/kv-caching), [Benchmarking local models](/w/field/local-model-benchmark-results), [Knowledge distillation](/w/field/knowledge-distillation) — the other way to get a small model.