4 results

for vllm

• PagedAttention: paging the KV cache like an OS pages memory
  PagedAttention is a memory-management scheme for serving LLMs, introduced in 2023 by Woosuk Kwon and colleagues (the vLLM paper). It stores the [KV cache](/w/field/kv-caching) used during autoregressive decoding in **fixed-size blocks mapped to non-contiguous physical memory**, b…
  field/paged-attention · paged-attention, kv-cache, inference, vllm, llm, memory
• Benchmarking local models: what 600+ graded runs showed
  |---|---|---|---|---|---| | gemma4 (vLLM) | 1 | **0.960** | 0.960 | 0.896 | 28.4 | | qwen3.8:27b | 1 | 0.949 | 0.949 | 0.882 | 29.5 |
  field/local-model-benchmark-results · benchmarks, evaluation, local-models, quantization, gguf, nvfp4, throughput, methodology
• Speculative decoding: a cheap model guesses, an expensive one signs off in one pass
  By 2024 it was standard in production serving: vLLM, TensorRT-LLM and SGLang ship it; Google uses it in AI Overviews; Apple, AWS and Meta have published extensions or deployments.
  field/speculative-decoding · speculative-decoding, inference, llm, latency, transformers
• Quantizing Qwen3.8-Flash-Next on one unified-memory box
  Field notes from fitting Qwen/Qwen3.8-Flash-Next (177.4 B params, 360 GB bf16) onto a single GB10-class machine — 121.7 GB unified memory, aarch64, CUDA 13, sm121. Measured with llm-compressor 0.13.0,
  field/qwen38-flash-next-on-one-unified-memory-gpu · quantization, nvfp4, llm-compressor, moe, unified-memory, vllm, offload, gb10, safetensors