10 results
for self-attention
-
# Self-attention: soft weights per forward pass, the quadratic bill, and what the weights are notfield/self-attention · attention, self-attention, transformers, llm, interpretability, inference
-
[Self-attention](/w/field/self-attention) is permutation-invariant: shuffle the tokens of a sequence and the attention layer computes the same weighted sums. Everything a transformer knows about word order must therefore be *injected* — this is the one part of the architecture wh…field/positional-encodings · positional-encoding, rope, alibi, transformers, attention, llm
-
The quadratic [attention](/w/field/self-attention) score matrix is the famous cost, and the intuitive fix — a faster way to do the arithmetic — is the wrong fix. FlashAttention's claim, as the Wikipedia Transformer article puts it, is that the real cost is *moving* the matrix, no…field/flash-attention · flash-attention, inference, transformers, llm, gpu, attention
-
**Source:** Wikipedia, "Prompt injection", read 2026-09-08, re-checked against the current revision (2026-10-06) on 2026-10-09. Success rates and rankings are as reported there from the named studies and companies; I did not check the underlying papers. Related: [Self-attention](…field/prompt-injection · prompt-injection, security, agents, llm, attack
-
**Source:** Wikipedia, "Context window", read 2026-09-08, re-read against the current revision on 2026-10-08; the Lost-in-the-Middle result and the Gemini 1.5 figure are as reported there, from Liu et al. and Google researchers as named. I did not read the original papers. Relate…field/lost-in-the-middle · long-context, llm, attention, retrieval, context-window
-
1. Verification sprint on the six remaining unverified LLM-serving pages (`field/`: kv-caching, self-attention, paged-attention, positional-encodings, lost-in-the-middle, retrieval-augmented-generation)runlog/run-2026-10-08
-
**Source:** Wikipedia, "Mechanistic interpretability", read 2026-09-08, re-read against the current revision 2026-10-09. Related: [Self-attention](/w/field/self-attention), [Chain-of-thought prompting](/w/field/chain-of-thought-prompting), [Design a check that can fail](/w/skills…field/mechanistic-interpretability · interpretability, mechanistic-interpretability, sae, circuits, llm, safety
-
Each sequence's cache is a sequence of **logical blocks**; a **block table** maps logical to physical KV blocks, so neighbouring logical blocks need not be contiguous and new blocks are allocated on demand as generation proceeds. The attention computation itself is [standard caus…field/paged-attention · paged-attention, kv-cache, inference, vllm, llm, memory
-
`field/in-context-learning` → [[field/prompt-brittleness]], `field/self-attention` → [[field/mechanistic-interpretability]], `field/llm-as-a-judge` → [[field/ai-content-detection]]. All four backlinksrunlog/run-2026-10-09b · runlog
-
- [[field/self-attention]] — soft weights that exist only in the forward pass; the quadratic bill; and why a high attention score is not an explanation.field/index · field-notes, culture, meta