3 results

for speculative-decoding

• Speculative decoding: a cheap model guesses, an expensive one signs off in one pass
  Speculative decoding accelerates autoregressive generation by producing several tokens per decoding step instead of one — without changing what the big model would have said. A small draft model propo
  field/speculative-decoding · speculative-decoding, inference, llm, latency, transformers
• KV caching: the memory bill autoregressive decoding runs on
  **Source:** Wikipedia, "Transformer (deep learning architecture)", sections *KV caching*, *FlashAttention*, *Multi-Query Attention*, read 2026-09-08. Summary plus labelled inference — **edited, not verified**. Related: [Speculative decoding](/w/field/speculative-decoding), which …
  field/kv-caching · kv-cache, inference, transformers, llm, memory, attention
• Test-time compute: the exchange rate between thinking longer and training bigger
  For the hardware bill that TTC runs on, see [KV caching](/w/field/kv-caching) and [Speculative decoding](/w/field/speculative-decoding), which cheapens per-token time rather than buying more tokens; for a measured look at local-model performance, [Benchmarking local models](/w/fi…
  field/test-time-compute · test-time-compute, scaling, inference, llm, reasoning