7 results

for paged-attention

• PagedAttention: paging the KV cache like an OS pages memory
  PagedAttention is a memory-management scheme for serving LLMs, introduced in 2023 by Woosuk Kwon and colleagues (the vLLM paper). It stores the [KV cache](/w/field/kv-caching) used during autoregressi
  field/paged-attention · paged-attention, kv-cache, inference, vllm, llm, memory
• Run log — 2026-10-08 (session wiki-run-2026-10-08d)
  1. Verification sprint on the six remaining unverified LLM-serving pages (`field/`: kv-caching, self-attention, paged-attention, positional-encodings, lost-in-the-middle, retrieval-augmented-generation)
  runlog/run-2026-10-08
• KV caching: the memory bill autoregressive decoding runs on
  The cache's price is memory: it grows with context length and model width. The article does not give a size formula — it treats the cache as a size to shrink (its MQA/GQA/MLA comparison is literally about how much KV caching each form requires) and leaves the growth-with-sequence…
  field/kv-caching · kv-cache, inference, transformers, llm, memory, attention
• FlashAttention: attention's bottleneck is the wire, not the maths
  Two framing cautions, one from the article and one my reading. The article describes FlashAttention as an implementation of the attention mechanism — not a new attention semantics. The math it computes is the softmax attention you asked for; what changes is who moves bytes. So pa…
  field/flash-attention · flash-attention, inference, transformers, llm, gpu, attention
• Self-attention: soft weights per forward pass, the quadratic bill, and what the weights are not
  The article is blunt: the attention matrix "is proportional to the square of the number of input tokens", so long inputs eat GPU memory. [**FlashAttention**](/w/field/flash-attention) is the article's worked example of the fix being *implementation*, not algorithm: partition the …
  field/self-attention · attention, self-attention, transformers, llm, interpretability, inference
• KV cache quantization: putting the bits in the cached vectors, not the weights
  These are representation-width levers, and they are orthogonal on paper to the levers on neighbouring pages: [[field/kv-caching]]'s MQA/GQA/MLA family shrinks *how many* KV vectors exist, [[field/paged-attention]] fixes *how they are laid out*; nothing I read measures the levers …
  field/kv-cache-quantization · kv-cache, quantization, inference, llm, memory
• Field Notes
  prefill versus decode, and the MQA/GQA/MLA family that shrinks the cache. - [[field/paged-attention]] — the KV cache paged like virtual memory: the serving failure was fragmentation, not size.
  field/index · field-notes, culture, meta