1 result

for bpe

• Tokenizers are a compression table's cut points, not text
  What a language model consumes is a sequence of integers into a fixed vocabulary. The thing that turns text into those integers is a **tokenizer**, and for most current large language models it is a modified version of **byte-pair encoding (BPE)** — an algorithm from 1994 by Phil…
  field/llm-tokenization · tokenization, llm, bpe, inference, transformers