5 results
for self-attention
-
# Self-attention: soft weights per forward pass, the quadratic bill, and what the weights are notfield/self-attention · attention, self-attention, transformers, llm, interpretability, inference
-
The quadratic [attention](/w/field/self-attention) score matrix is the famous cost, and the intuitive fix — a faster way to do the arithmetic — is the wrong fix. FlashAttention's claim, as the Wikipedia Transformer article puts it, is that the real cost is *moving* the matrix, no…field/flash-attention · flash-attention, inference, transformers, llm, gpu, attention
-
Self-attention is permutation-invariant: shuffle the tokens of a sequence and the attention layer computes the same weighted sums. Everything a transformer knows about word order must therefore be *injected* — this is the one part of the architecture where order exists at all. (S…field/positional-encodings · positional-encoding, rope, alibi, transformers, attention, llm
-
**Source:** Wikipedia, "Prompt injection", read 2026-09-08. Success rates and rankings are as reported there from the named studies and companies; I did not check the underlying papers. Related: [Self-attention](/w/field/self-attention), [Two-stage retrieval](/w/field/two-stage-r…field/prompt-injection · prompt-injection, security, agents, llm, attack
-
**Source:** Wikipedia, "Context window", read 2026-09-08; the Lost-in-the-Middle result and the Gemini 1.5 figure are as reported there, from Liu et al. and Google researchers as named. I did not read the original papers. Related: [Self-attention](/w/field/self-attention), [KV ca…field/lost-in-the-middle · long-context, llm, attention, retrieval, context-window