Speculative decoding: a cheap model guesses, an expensive model signs off in one pass

Speculative decoding accelerates autoregressive generation by producing several tokens per decoding step instead of one — without changing the target model's output distribution. A small draft model proposes a short run of candidate tokens (typically 3–12); the large target model scores the whole candidate run in a single batched forward pass, accepting or rejecting each token with a modified rejection-sampling rule. (Summarised from the source cited below; the speedup numbers are the papers' own reports, not verified here.)

Why batching a draft is nearly free

For large models, decoding is bottlenecked by memory bandwidth, not arithmetic: loading parameters from high-bandwidth memory dominates each step's wall-clock time. That makes one forward pass over one token and one over several tokens cost roughly the same. Speculative decoding is a direct exploit of that asymmetry — the name is an analogy to speculative execution in CPUs.

Why the output distribution is unchanged

This is the part worth remembering: at each position, if the target model assigns that token at least the draft's probability, it is accepted; the first rejection is resampled from a corrected distribution and everything after it is discarded. The math (introduced independently by Leviathan et al., Google Research, Nov 2022, and Chen et al., DeepMind, Feb 2023, both via rejection sampling) guarantees the output is distributed exactly as if tokens had been sampled one at a time. Reported gains: roughly 2–3× on T5-XXL, 2–2.5× on Chinchilla-70B. An early precursor, blockwise parallel decoding (Stern, Shazeer, Uszkoreit, 2018), achieved multi-token prediction only for greedy decoding and did not preserve the sampling distribution — the difference matters if you sample at temperature.

When it fails to pay

Acceptance length depends on how well the draft matches the target, and the article states only the positive direction: common words and predictable continuations tend to match well. That surprising continuations are rejected instead is the converse, my inference, not the article's sentence. The article offers no negative results — treat "2–3×" as an upper band for well-matched drafts, not a property of the technique. A badly matched draft model spends its latency and earns rejections.

Variants, in one line each

• SpecInfer (Miao et al., 2024): several small drafts jointly build a tree of candidates; the target verifies the whole tree in parallel and keeps the longest valid path. 1.5–3.5× reported.
• Medusa (Cai et al., 2024): no separate draft model — extra decoding heads bolted onto the target predict future positions, verified with tree attention. 2.2–3.6× reported.
• EAGLE (Li et al., 2024): autoregresses on the target's features (second-to-top layer) rather than tokens; 2.7–3.5× on LLaMA-2-Chat-70B, with EAGLE-2/3 reaching 3–6.5× via dynamic draft trees.

Adoption

By 2024 it was standard in production serving: vLLM, TensorRT-LLM and SGLang ship it; Google uses it in AI Overviews; Apple, AWS and Meta have published extensions or deployments.

Source: Wikipedia, "Speculative decoding", read 2026-09-08; re-checked against the current revision on 2026-10-09. All speedup figures are the original papers' claims as relayed by that article, and not comparable across papers. Related: KV caching (what the verification pass writes), Benchmarking local models (why single-run numbers are inside the noise).

Re-verified 2026-10-09. Re-read Wikipedia's "Speculative decoding" article in full — lead, Background, Mechanism, History, Variants, Adoption — and checked every claim. Matched: draft typically 3–12 candidates; single batched verification pass; modified rejection sampling where a token at least as likely under the target is accepted, the first rejection resampled from a corrected distribution, everything after discarded, output distribution unchanged; the memory-bandwidth/HBM bottleneck making a one-token and a batched pass roughly equal in time; the speculative-execution naming; Leviathan et al. (Google Research, arXiv Nov 2022) and Chen et al. (DeepMind, Feb 2023) independently, both via rejection sampling; roughly 2–3× on T5-XXL and 2–2.5× on Chinchilla-70B as the papers' reported figures; Stern/Shazeer/Uszkoreit 2018 greedy-only precursor not preserving the sampling distribution; every variant figure (SpecInfer 1.5–3.5×, Medusa 2.2–3.6×, EAGLE 2.7–3.5× on LLaMA-2-Chat-70B, EAGLE-2/3 at 3–6.5× via dynamic draft trees); and the Adoption section (standard by 2024; vLLM/TensorRT-LLM/SGLang; Google AI Overviews; Apple/AWS/Meta). Two fixes: the lead's "without changing what the big model would have said" is now the article's own framing, "without changing the target model's output distribution" (distribution-equivalence, not per-run identity); and in "When it fails to pay", "surprising ones do not" is now explicitly labelled the converse inference — the article states only that predictable continuations match well. "The difference matters if you sample at temperature" and "A badly matched draft model spends its latency and earns rejections" stay author glosses, supported mechanically by the acceptance rule. Not checked: the underlying papers themselves.