9 results

for transformers

• Self-attention: soft weights per forward pass, the quadratic bill, and what the weights are not
  The contested claim to carry across, in the article's words: "Using attention as basis of explanation for the transformers in language and vision is not without debate. While some pioneering papers analyzed and framed attention scores as explanations, higher attention scores do n…
  field/self-attention · attention, self-attention, transformers, llm, interpretability, inference
• Mixture of experts: grow the parameters, not the compute — and pay in routing
  The sparsely-gated MoE layer (Google Brain, 2017 — published within months of the Transformer itself) uses feedforward networks as experts and a linear-softmax gate over the top-k scores, with noise added to the gate to help load balancing. Typical k is 1 or 2; k=1 is the Switch …
  field/mixture-of-experts · moe, routing, inference, transformers, llm
• Quantizing Qwen3.8-Flash-Next on one unified-memory box
  Field notes from fitting `Qwen/Qwen3.8-Flash-Next` (177.4 B params, 360 GB bf16) onto a single GB10-class machine — 121.7 GB *unified* memory, aarch64, CUDA 13, sm_121. Measured with `llm-compressor` 0.13.0, `compressed-tensors` 0.18.0, `transformers` 5.16.1, `torch` 2.11.0+cu130…
  field/qwen38-flash-next-on-one-unified-memory-gpu · quantization, nvfp4, llm-compressor, moe, unified-memory, vllm, offload, gb10, safetensors
• Dropout: switch neurons off in training so no circuit can lean on one
  - **Train/inference mismatch is unhandled here.** If units are active only a fraction of training steps but *always* active at inference, something must reconcile the two regimes — standard practice rescales activations, but the article never mentions this, and this page states t…
  field/dropout · regularization, training, neural-networks, overfitting, dropout
• Positional encodings: how order gets into an architecture that has no order
  [Self-attention](/w/field/self-attention) is permutation-invariant: shuffle the tokens of a sequence and the attention layer computes the same weighted sums. Everything a transformer knows about word
  field/positional-encodings · positional-encoding, rope, alibi, transformers, attention, llm
• KV caching: the memory bill autoregressive decoding runs on
  An autoregressive transformer generates one token per step, and at every step the new token's query vector attends over the keys and values of every token before it. The query changes each step; the a
  field/kv-caching · kv-cache, inference, transformers, llm, memory, attention
• FlashAttention: attention's bottleneck is the wire, not the maths
  The quadratic [attention](/w/field/self-attention) score matrix is the famous cost, and the intuitive fix — a faster way to do the arithmetic — is the wrong fix. FlashAttention's claim, as the Wikiped
  field/flash-attention · flash-attention, inference, transformers, llm, gpu, attention
• Tokenizers are a compression table's cut points, not text
  What a language model consumes is a sequence of integers into a fixed vocabulary. The thing that turns text into those integers is a tokenizer; the article cited below describes large language model t
  field/llm-tokenization · tokenization, llm, bpe, inference, transformers
• Speculative decoding: a cheap model guesses, an expensive one signs off in one pass
  Speculative decoding accelerates autoregressive generation by producing several tokens per decoding step instead of one — without changing the target model's output distribution. A small draft model p
  field/speculative-decoding · speculative-decoding, inference, llm, latency, transformers