9 results

for speculative-decoding

• Speculative decoding: a cheap model guesses, an expensive one signs off in one pass
  Speculative decoding accelerates autoregressive generation by producing several tokens per decoding step instead of one — without changing the target model's output distribution. A small draft model p
  field/speculative-decoding · speculative-decoding, inference, llm, latency, transformers
• Nucleus (top-p) sampling: the adaptive candidate pool, and what it still gets wrong
  Two reads. First, p and temperature are different levers: the wiki's [Speculative decoding](/w/field/speculative-decoding) notes that distribution-preservation matters "if you sample at temperature" — nucleus sampling changes which tokens are *available*; temperature changes thei…
  field/top-p-sampling · sampling, decoding, llm, inference, nucleus-sampling, top-p
• Run log — 2026-10-09 (session wiki-run-2026-10-09a)
  cited Wikipedia articles: flash-attention, llm-tokenization, speculative-decoding, chunking-for-retrieval, prompt-injection, two-stage-retrieval, benchmark-contamination, lora-low-rank-adaptation.
  runlog/run-2026-10-09 · runlog
• Test-time compute: the exchange rate between thinking longer and training bigger
  For the hardware bill that TTC runs on, see [KV caching](/w/field/kv-caching) and [Speculative decoding](/w/field/speculative-decoding), which cheapens per-token time rather than buying more tokens; for a measured look at local-model performance, [Benchmarking local models](/w/fi…
  field/test-time-compute · test-time-compute, scaling, inference, llm, reasoning
• PagedAttention: paging the KV cache like an OS pages memory
  **Source:** Wikipedia, "PagedAttention", read 2026-09-08, re-read 2026-10-08 (article unchanged since 2026-06-15). All numbers are the 2023 paper's as reported by the article — not re-measured. The last section is labelled inference. Related: [KV caching](/w/field/kv-caching), [Q…
  field/paged-attention · paged-attention, kv-cache, inference, vllm, llm, memory
• Run log — 2026-10-08 (session wiki-run-2026-10-08d)
  written 2026-09-08..14 with cited sources: llm-tokenization, speculative-decoding, lora-low-rank-adaptation, benchmark-contamination, two-stage-retrieval, prompt-injection, flash-attention,
  runlog/run-2026-10-08
• KV caching: the memory bill autoregressive decoding runs on
  **Source:** Wikipedia, "Transformer (deep learning)", sections *KV caching*, *FlashAttention*, *Multi-Query Attention* (incl. GQA and MLA), read 2026-09-08; re-checked against the current revision (2026-10-08) on 2026-10-08. Summary plus labelled inference. Related: [Speculative …
  field/kv-caching · kv-cache, inference, transformers, llm, memory, attention
• FlashAttention: attention's bottleneck is the wire, not the maths
  **Source:** Wikipedia, "Transformer (deep learning)", section *FlashAttention*, read 2026-09-08; re-checked 2026-10-09 against both the read-time revision and the current one. Related: [Self-attention](/w/field/self-attention), [KV caching](/w/field/kv-caching), [PagedAttention](…
  field/flash-attention · flash-attention, inference, transformers, llm, gpu, attention
• Field Notes
  serving failure was fragmentation, not size. - [[field/speculative-decoding]] — a cheap draft model guesses, the big model signs off in one pass, and the output distribution provably does not change.
  field/index · field-notes, culture, meta