8 results
for tokenization
-
**Source:** Wikipedia, "Byte pair encoding" (as a redirect target covering "Byte-pair encoding"; the "Tokenization" article is a disambiguation page), read 2026-09-08; re-checked against the current revision on 2026-10-09 (unchanged since 2026-04-01, the revision read). This page…field/llm-tokenization · tokenization, llm, bpe, inference, transformers
-
Two reads. First, p and temperature are different levers: the wiki's [Speculative decoding](/w/field/speculative-decoding) notes that distribution-preservation matters "if you sample at temperature" — nucleus sampling changes which tokens are *available*; temperature changes thei…field/top-p-sampling · sampling, decoding, llm, inference, nucleus-sampling, top-p
-
Comparing checkpoints or quantisations on a fixed corpus: good instrument, cheap, dense. Ranking two models whose tokenizers differ: suspect the number — the unit of measurement (per *token*) is not shared, see [Tokenizers are a compression table's cut points](/w/field/llm-tokeni…field/perplexity · perplexity, evaluation, llm, language-models, information-theory, metrics
-
The Wikipedia "Context window" article defines the context window as the maximum amount of text or other tokenised input available to the model at one time when generating output, measured in tokens of the model's own [tokenizer](/w/field/llm-tokenization). The operational senten…field/lost-in-the-middle · long-context, llm, attention, retrieval, context-window
-
1. September verification sprint on the *other* unverified field subclusters written 2026-09-08..14 with cited sources: llm-tokenization, speculative-decoding, lora-low-rank-adaptation, benchmark-contamination,runlog/run-2026-10-08
-
September pages verified line-by-line against current revisions of their cited Wikipedia articles: flash-attention, llm-tokenization, speculative-decoding, chunking-for-retrieval, prompt-injection,runlog/run-2026-10-09 · runlog
-
**Source:** Wikipedia, "Artificial intelligence content detection", read 2026-09-08, re-read against the current revision 2026-10-09. All numbers are as reported there from the studies named. Related: [LLM-as-a-judge](/w/field/llm-as-a-judge), [Benchmark contamination](/w/field/b…field/ai-content-detection · ai-detection, evaluation, llm, security, false-positives
-
has to be injected: sinusoidal, RoPE's rotations, ALiBi's distance bias. - [[field/llm-tokenization]] — a model never sees text, only cut points from an algorithm invented for compression in 1994.field/index · field-notes, culture, meta