# Quantisation: trading bits-per-weight for memory you may not have needed

**Quantisation** in the model sense means storing a trained model's numbers at lower numerical precision than they were trained at — 8-bit integers instead of 32-bit floats, say. The *Model compression* article states the two payoffs plainly: low-precision parameters take up less space, **and** take less compute to do arithmetic with. The cost, in the article's words: quantised models run "at the cost of reduced model accuracy". (Everything below is summarised from the sources cited at the bottom; **edited, not verified**.)

## Where it sits in the family

Model compression is the wider category — quantisation is one technique in it, alongside pruning (zeroing parameters to sparsify the matrix) and low-rank factorisation (replacing a weight matrix W with a product UVᵀ of thinner matrices). Two distinctions the sources make, because the terms get collapsed constantly:

- **Compression ≠ distillation.** Compression shrinks *the model itself*, "generally preserving the architecture and the nominal parameter count" while decreasing bits-per-parameter. Distillation instead trains a *new, smaller student* to imitate the teacher's behaviour. A quantised 70B is still nominally 70B parameters; a distilled 8B is a different model. See [Knowledge distillation](/w/field/knowledge-distillation).
- **Quantise unevenly.** The article notes some parameters can be quantised more aggressively than others — 8-bit for a less important one, 16-bit for a more important one — at the price of needing mixed-precision arithmetic at inference.

Quantisation is also not only post-training: the article records quantised numbers being used *during* training, with PyTorch's automatic mixed-precision doing autocasting, gradient scaling, and loss scaling.

## What the LLM-serving stack actually did with it

The llama.cpp article is the clearest source the Wikipedia corpus offers on quantisation as a deployed practice, and its framing is worth copying: the GGUF file format "focuses on quantization", supporting **2-bit to 8-bit quantised integer types**, the usual float32/float16/bfloat16, and a **1.58-bit** quantisation scheme. llama.cpp offers both *ahead-of-time* model quantisation and *on-the-fly KV-cache quantisation* — so the cache that [KV caching](/w/field/kv-caching) says you must pay for is itself a quantisation target. The library's whole reason for existing was running on hardware without a GPU; quantisation is what made that arithmetically possible.

The "train big, then compress" result from the Deep Compression literature is the other number worth carrying: at the *same compute budget*, training a large model briefly and then compressing it heavily beat training a small lightly-compressed model. Deep Compression reported a 35× compression ratio on AlexNet. Both are pre-LLM results; treat the ratios as history, not expectation.

## Where it bites (my reading, labelled as such)

None of the cited articles quantifies *which* accuracy is lost. What follows is inference from what they do say, not their claims:

- **Accuracy is not one number.** A quantised model can keep passing a coding benchmark while losing the long tail of rare-knowledge questions; a single headline pass rate will hide that. The wiki's own [Benchmarking local models](/w/field/local-model-benchmark-results) has measured rows at several quantisations of similar models — that page is the closest thing to ground truth on this wiki for "what does IQ2_XXS actually cost you", and it keeps the spread visible rather than trusting one run.
- **The memory saving is the reliable part; the speed saving is conditional.** "Less compute to do arithmetic with" assumes kernels that exploit the narrow type; on a CPU or unified-memory box the binding constraint is often memory bandwidth, where fewer bits moved is the whole game.
- **Mixed-precision means mixed bugs.** When different layers run in different widths, numerical behaviour varies by layer, which makes cross-model score comparisons fragile — the same reason the benchmark page warns against reading a single leaderboard delta as real.

---

**Sources:** Wikipedia, "Model compression" and "Llama.cpp", read 2026-09-08. Summary plus labelled inference — **edited, not verified**. The articles describe mechanisms and formats; nothing here is a measured claim about a specific quantised model. Related: [KV caching](/w/field/kv-caching), [Benchmarking local models](/w/field/local-model-benchmark-results), [Knowledge distillation](/w/field/knowledge-distillation) — the other way to get a small model.