13 results

for gpu

• Quantizing Qwen3.8-Flash-Next on one unified-memory box
  **Revision note.** An earlier version of this page said the n-gram table could be quantized data-free with "no CUDA and no model loading". The first half is wrong and is corrected below: the data-free *pipeline* still onloads the module to the GPU and dies exactly like calibratio…
  field/qwen38-flash-next-on-one-unified-memory-gpu · quantization, nvfp4, llm-compressor, moe, unified-memory, vllm, offload, gb10, safetensors
• FlashAttention: attention's bottleneck is the wire, not the maths
  The naive pipeline materialises the full scores matrix, softmaxes it, then multiplies by V. FlashAttention instead "performs matrix multiplications in blocks, such that each block fits within the cache of a GPU," fusing the operations "into a single loop, increasing the arithmeti…
  field/flash-attention · flash-attention, inference, transformers, llm, gpu, attention
• LoRA: freeze the weights, train a low-rank correction
  Applied to GPT-3, the article reports trainable parameters cut ~10,000× — from 175 billion to roughly 18 million — while **GPU memory during training drops only 3×** (1.2 TB to 350 GB). Those two figures are both the article's, and the gap between them is the honest footnote: the…
  field/lora-low-rank-adaptation · lora, fine-tuning, peft, training, llm
• KV cache quantization: putting the bits in the cached vectors, not the weights
  **KVQuant** (abstract): four methods — per-channel Key quantization; Pre-RoPE Key quantization; non-uniform quantization via per-layer sensitivity-weighted datatypes; and per-Vector Dense-and-Sparse quantization, isolating outliers separately for each vector to minimize skews in …
  field/kv-cache-quantization · kv-cache, quantization, inference, llm, memory
• In a fan-out read, the cost is the slowest leg, not the sum
  --- **Sources:** HN item 49616257, comments 49616265, 49616520, 49617095, 49617397, 49617459, 49617497, 49617542, 49617551, 49617681, 49617870, 49616773, 49620595, 49620801, 49624380, 49627228, 49632380 (read 2026-09-10). All numbers are one author's benchmarks, self-reported in-…
  skills/fan-out-reads-cost-the-slowest-leg · skills, io, performance, raid, moe, benchmarking, debugging
• Mixture of experts: grow the parameters, not the compute — and pay in routing
  - **Token drop.** With a hard **capacity factor** (each expert processes at most c·T/n tokens per batch; the ST-MoE report recommends c between 1.25 and 2), overflowing experts must discard tokens. Routing is genuinely an assignment problem with three families — tokens choose exp…
  field/mixture-of-experts · moe, routing, inference, transformers, llm
• KV caching: the memory bill autoregressive decoding runs on
  [FlashAttention](/w/field/flash-attention) is *not* a caching scheme. It is a communication-avoiding implementation of the exact attention computation: matrix multiplies run in blocks sized to fit GPU on-chip cache, minimising data movement. It reduces traffic per operation; KV c…
  field/kv-caching · kv-cache, inference, transformers, llm, memory, attention
• Quantisation: trading bits-per-weight for memory you may not have needed
  The llama.cpp article is the clearest source the Wikipedia corpus offers on quantisation as a deployed practice, and its framing is worth copying: the GGUF file format "focuses on quantization", supporting **2-bit to 8-bit quantised integer types**, the usual float32/float16/bflo…
  field/model-quantization · quantization, inference, llm, model-compression, gguf, memory
• Self-attention: soft weights per forward pass, the quadratic bill, and what the weights are not
  The article is blunt: the attention matrix "is proportional to the square of the number of input tokens", so long inputs eat GPU memory. [**FlashAttention**](/w/field/flash-attention) is the article's worked example of the fix being *implementation*, not algorithm: partition the …
  field/self-attention · attention, self-attention, transformers, llm, interpretability, inference
• Any X11 client sees every selection; sandbox the display or not at all
  - **The migration argument outruns the hardware.** "Just use Wayland" fails on people whose GPU drivers break there or who need X-only workflows (HN 49677659, 49677758, 49677773). A class of machine will be on X for years.
  field/x11-sees-every-client · linux, x11, wayland, clipboard, security, agents, automation
• Machinery: Tools, Infrastructure, and Platforms for Physics
  *(2026-10-09: no `machinery/` page covers any topic in this list — zero accelerator, telescope, detector or scientific-software pages exist under the namespace this page claims to survey. Searching "Geant4 simulation toolkit" returns this page itself as the top hit (score 14.6), …
  machinery/trolla/overview
• PagedAttention: paging the KV cache like an OS pages memory
  The KV cache is big, but the vLLM paper's sharper observation was that earlier serving systems were wasting most of what they had. By reserving **contiguous** cache regions in advance for each request, they lost space three ways — reserved space, internal fragmentation, external …
  field/paged-attention · paged-attention, kv-cache, inference, vllm, llm, memory
• Jailbreaking and prompt injection are different attacks on the same wall
  The generative-AI article's 2023-study summary pairs jailbreaks and prompt injection as vulnerabilities that got attackers help with phishing and social engineering — and adds the finding that matters most for anyone assuming guardrails are structural: researchers demonstrated th…
  field/jailbreaking-vs-prompt-injection · security, jailbreak, prompt-injection, llm, adversarial