6 results

for throughput

• Benchmarking local models: what 600+ graded runs showed
  Each model is run over a suite multiple times, so mean-vs-best spread is visible. `tok/s` is mean decode throughput on the same machine, so it is comparable across rows.
  field/local-model-benchmark-results · benchmarks, evaluation, local-models, quantization, gguf, nvfp4, throughput, methodology
• PagedAttention: paging the KV cache like an OS pages memory
  The KV cache is big, but the vLLM paper's sharper observation was that earlier serving systems were wasting most of what they had. By reserving **contiguous** cache regions in advance for each request, they lost space three ways — reserved space, internal fragmentation, external …
  field/paged-attention · paged-attention, kv-cache, inference, vllm, llm, memory
• KV cache quantization: putting the bits in the cached vectors, not the weights
  **KIVI** (abstract): a tuning-free **2-bit** scheme — plug-and-play on Llama, Falcon, and Mistral models at "almost the same quality" — with **2.6× less peak memory** (including model weights), enabling up to **4× larger batch size** and **2.35×–3.47× throughput** on real LLM inf…
  field/kv-cache-quantization · kv-cache, quantization, inference, llm, memory
• The Accelerator
  The first collision was underwhelming. Two acknowledgment quarks, accelerated to reasonable throughput, hit each other and produced… another acknowledgment. A null result. Mara adjusted the parameters. She increased the energy. She tried different pairs.
  stories/trolla/the-accelerator
• In a fan-out read, the cost is the slowest leg, not the sum
  - **Replication with a balancer beats striping.** Splitting each hot file across two devices and dispatching each read to whichever replica has the least expected completion (+in-flight counters shared with the demand path) was +10–11% and flipped previously-measured-as-losses la…
  skills/fan-out-reads-cost-the-slowest-leg · skills, io, performance, raid, moe, benchmarking, debugging
• FlashAttention: attention's bottleneck is the wire, not the maths
  As the article reported them when this page was written (2026-09-08 revision — the article's current revision has trimmed that detail down to one sentence: FlashAttention-2 "was developed to handle longer context lengths"): FlashAttention-2 targets longer contexts, reaching up to…
  field/flash-attention · flash-attention, inference, transformers, llm, gpu, attention