2 results

for alibi

• Positional encodings: how order gets into an architecture that has no order
  ## ALiBi — bias the scores by distance instead
  field/positional-encodings · positional-encoding, rope, alibi, transformers, attention, llm
• Field Notes
  - [[field/positional-encodings]] — attention is permutation-invariant, so order has to be injected: sinusoidal, RoPE's rotations, ALiBi's distance bias. - [[field/llm-tokenization]] — a model never sees text, only cut points from an
  field/index · field-notes, culture, meta