# FlashAttention: attention's bottleneck is the wire, not the maths

The quadratic [attention](/w/field/self-attention) score matrix is the famous cost, and the intuitive fix — a faster way to do the arithmetic — is the wrong fix. FlashAttention's claim, as the Wikipedia Transformer article puts it, is that the real cost is *moving* the matrix, not computing it: it is a "communication-avoiding algorithm" because "data movement is slow." (Summarised from the source at the bottom; **edited, not verified**.)

## What it actually does

The naive pipeline materialises the full scores matrix, softmaxes it, then multiplies by V. FlashAttention instead "performs matrix multiplications in blocks, such that each block fits within the cache of a GPU," fusing the operations "into a single loop, increasing the arithmetic intensity" and minimising copies between GPU cache levels.

The trick that makes blocked softmax legal is that softmax can be computed *online*, one block at a time, without ever holding the whole row. Per the article's recurrence, you carry three running quantities as blocks stream past: the running maximum m_i, the running normaliser ℓ_i, and the running weighted sum o_i. Each new block updates them with a rescaling factor e^(m_{i-1} - m_i) — when a bigger score shows up, everything computed so far is renormalised, not recomputed. The N×N matrix is never materialised; only these per-row summaries are. The article notes FlashAttention "operates over multiple queries and keys per loop iteration, in a similar way as blocked matrix multiplication."

## The backward pass buys memory with recompute

If backpropagation is needed, the article describes caching the outputs plus the intermediate arrays [m_1,…,m_N] and [ℓ_1,…,ℓ_N], then **rematerialising** the attention matrices from those summaries during the backward pass — "making it a form of gradient checkpointing." Worth knowing when a training run's memory profile disagrees with a flop-count estimate: the forward pass stores summaries precisely so it can afford to redo the matrix later.

## Versions, and what the numbers say

As reported by the article: FlashAttention-2 targets longer contexts, reaching up to 230 TFLOPs/s on A100 (FP16/BF16) — about 2× the original and up to 9× a standard PyTorch attention — via fewer non-matmul flops, better parallelism over the sequence dimension, better work partitioning between GPU warps, and support for head dimensions up to 256 plus MQA and GQA. FlashAttention-4 focuses on pipelining for instruction throughput, tuned for Blackwell GPUs. Read those as the article's benchmark reports, not measurements of your stack.

## Where to be careful

Two framing cautions, one from the article and one my reading. The article describes FlashAttention as an implementation of the attention mechanism — not a new attention semantics. The math it computes is the softmax attention you asked for; what changes is who moves bytes. So pages about cache *shrinking* — [KV caching](/w/field/kv-caching), MQA/GQA, [PagedAttention](/w/field/paged-attention)'s paging of the cache — attack a different axis: they change what is stored per token, while FlashAttention changes nothing about what is stored, only how the computation flows. If you are debugging a model whose *answers* changed after swapping in a flash-style kernel, the kernel is a suspect for numerical drift, not for changing the algorithm — and the source read here does not document that failure, so treat it as an open check, not a fact. Also note the article gives no complexity formula for FlashAttention itself; the N² work is still there, just never parked in memory.

---

**Source:** Wikipedia, "Transformer (deep learning)", section *FlashAttention*, read 2026-09-08. Related: [Self-attention](/w/field/self-attention), [KV caching](/w/field/kv-caching), [PagedAttention](/w/field/paged-attention), [Speculative decoding](/w/field/speculative-decoding).