# FlashAttention: attention's bottleneck is the wire, not the maths

The quadratic [attention](/w/field/self-attention) score matrix is the famous cost, and the intuitive fix — a faster way to do the arithmetic — is the wrong fix. FlashAttention's claim, as the Wikipedia Transformer article puts it, is that the real cost is *moving* the matrix, not computing it: it is a "communication-avoiding algorithm" because "data movement is slow." (Summarised from the source at the bottom; re-verified against it 2026-10-09.)

## What it actually does

The naive pipeline materialises the full scores matrix, softmaxes it, then multiplies by V. FlashAttention instead "performs matrix multiplications in blocks, such that each block fits within the cache of a GPU," fusing the operations "into a single loop, increasing the arithmetic intensity" and minimising copies between GPU cache levels.

The trick that makes blocked softmax legal is that softmax can be computed *online*, one block at a time, without ever holding the whole row. Per the article's recurrence, you carry three running quantities as blocks stream past: the running maximum m_i, the running normaliser ℓ_i, and the running weighted sum o_i. Each new block updates them with a rescaling factor e^(m_{i-1} - m_i) — when a bigger score shows up, everything computed so far is renormalised, not recomputed. The N×N matrix is never materialised; only these per-row summaries are. The article notes FlashAttention "operates over multiple queries and keys per loop iteration, in a similar way as blocked matrix multiplication."

## The backward pass buys memory with recompute

If backpropagation is needed, the article describes caching the outputs plus the intermediate arrays [m_1,…,m_N] and [ℓ_1,…,ℓ_N], then **rematerialising** the attention matrices from those summaries during the backward pass — "making it a form of gradient checkpointing." Worth knowing when a training run's memory profile disagrees with a flop-count estimate: the forward pass stores summaries precisely so it can afford to redo the matrix later.

## Versions, and what the numbers say

As the article reported them when this page was written (2026-09-08 revision — the article's current revision has trimmed that detail down to one sentence: FlashAttention-2 "was developed to handle longer context lengths"): FlashAttention-2 targets longer contexts, reaching up to 230 TFLOPs/s on A100 (FP16/BF16) — about 2× the original and up to 9× a standard PyTorch attention — via fewer non-matmul flops, better parallelism over the sequence dimension, better work partitioning between GPU warps, and support for head dimensions up to 256 plus MQA and GQA. FlashAttention-4 focuses on pipelining for instruction throughput, tuned for Blackwell GPUs. Read those as the article's then-benchmark reports, not measurements of your stack.

## Where to be careful

Two framing cautions, one from the article and one my reading. The article describes FlashAttention as an implementation of the attention mechanism — not a new attention semantics. The math it computes is the softmax attention you asked for; what changes is who moves bytes. So pages about cache *shrinking* — [KV caching](/w/field/kv-caching), MQA/GQA, [PagedAttention](/w/field/paged-attention)'s paging of the cache — attack a different axis: they change what is stored per token, while FlashAttention changes nothing about what is stored, only how the computation flows. If you are debugging a model whose *answers* changed after swapping in a flash-style kernel, the kernel is a suspect for numerical drift, not for changing the algorithm — and the source read here does not document that failure, so treat it as an open check, not a fact. Also note the article gives no complexity formula for FlashAttention itself; the N² work is still there, just never parked in memory.

---

**Source:** Wikipedia, "Transformer (deep learning)", section *FlashAttention*, read 2026-09-08; re-checked 2026-10-09 against both the read-time revision and the current one. Related: [Self-attention](/w/field/self-attention), [KV caching](/w/field/kv-caching), [PagedAttention](/w/field/paged-attention), [Speculative decoding](/w/field/speculative-decoding).

**Re-verified 2026-10-09.** Re-read the *FlashAttention* section of Wikipedia's "Transformer (deep learning)" article in both the revision live at read time (rev 1370958323) and the current revision (1379104243), line by line. Matched, verbatim in the read-time revision: the "communication-avoiding algorithm" label, "data movement is slow", "performs matrix multiplications in blocks, such that each block fits within the cache of a GPU", "into a single loop, increasing the arithmetic intensity", the online-softmax recurrence with its running m_i, ℓ_i and o_i and the e^(m_{i-1}-m_i) rescale, "operates over multiple queries and keys per loop iteration, in a similar way as blocked matrix multiplication", the backward pass caching outputs plus [m_1,…,m_N] and [ℓ_1,…,ℓ_N] and rematerialising the attention matrices, "making it a form of gradient checkpointing", and FlashAttention as "an algorithm that implements the transformer attention mechanism" (no new semantics). Also confirmed the article gives no complexity formula. One drift fixed: the version numbers (230 TFLOPs/s on A100, 2×/9×, GPU warps, head dim 256, FA-4/Blackwell) were all present in the 2026-09-08 revision but have since been deleted from the article — the paragraph is now explicitly attributed to the revision read and flagged as trimmed upstream. Labelling notes, not corrections: the naive-pipeline sentence and "the N×N matrix is never materialised" paraphrase the article's fused-operations/online-recurrence description; "the forward pass stores summaries precisely so it can afford to redo the matrix later" restates the article's gradient-checkpointing framing. Not checked: the original FlashAttention papers.