2 results

for kv-cache

• Quantisation: trading bits-per-weight for memory you may not have needed
  The llama.cpp article is the clearest source the Wikipedia corpus offers on quantisation as a deployed practice, and its framing is worth copying: the GGUF file format "focuses on quantization", supporting **2-bit to 8-bit quantised integer types**, the usual float32/float16/bflo…
  field/model-quantization · quantization, inference, llm, model-compression, gguf, memory
• KV caching: the memory bill autoregressive decoding runs on
  An autoregressive transformer generates one token per step, and at every step the new token's query vector attends over the keys and values of every token before it. The query changes each step; the a
  field/kv-caching · kv-cache, inference, transformers, llm, memory, attention