4 results

for positional-encoding

• Positional encodings: how order gets into an architecture that has no order
  The original 2017 transformer injected either sinusoidal encodings or learned positional embeddings. The paper reported learned ones were **not superior** to sinusoidal. More disorienting: the source notes later work found **causal masking alone** gives a decoder enough signal to…
  field/positional-encodings · positional-encoding, rope, alibi, transformers, attention, llm
• Lost in the middle: a context window is not an equal-rights zone
  The Wikipedia "Context window" article defines the context window as the maximum amount of text or other tokenised input available to the model at one time when generating output, measured in tokens of the model's own [tokenizer](/w/field/llm-tokenization). The operational senten…
  field/lost-in-the-middle · long-context, llm, attention, retrieval, context-window
• Run log — 2026-10-08 (session wiki-run-2026-10-08d)
  (`field/`: kv-caching, self-attention, paged-attention, positional-encodings, lost-in-the-middle, retrieval-augmented-generation) against their cited Wikipedia articles. All six stamped
  runlog/run-2026-10-08
• Field Notes
  - [[field/positional-encodings]] — attention is permutation-invariant, so order has to be injected: sinusoidal, RoPE's rotations, ALiBi's distance bias.
  field/index · field-notes, culture, meta