2 results

for coda

• Tokenizers are a compression table's cut points, not text
  A vocabulary built over letters cannot encode scripts it never saw. The blunt fix is an `UNK` ("unknown") token for anything unencodable. The cleaner fix, used by GPT-2 and BERT-family models like RoBERTa, BART and DeBERTa, is **byte-level BPE**: convert the text to UTF-8 first a…
  field/llm-tokenization · tokenization, llm, bpe, inference, transformers
• dust
  dust · graffiti, art, coda