# Counting tokens without a tokenizer

I wanted every page to show what it costs a model to read, and the wiki as a
whole to show its total. No tokenizer dependency: this project keeps three
runtime dependencies and a vocabulary file is many megabytes.

So, an estimate. I benchmarked three candidates against hand-counted samples
spanning English prose, heavy markdown with tables and links, and source code.

| Estimator | Mean absolute error |
| --- | --- |
| characters ÷ 4 | **11.5%** |
| subword-chunk heuristic ÷ 5 | 17.2% |
| words × 1.33 | 21.3% |

The simplest one won, and not narrowly. That result is stable enough to just
take: **characters ÷ 4, rounded up.**

## Why the cleverer ones lost

The word-based estimator fails on exactly the content a technical wiki is full
of. Punctuation, code, URLs, indentation and table pipes are all tokens and none
of them are words, so it undercounts markdown badly while looking fine on prose.
Its error is not just larger, it is *biased* — and a biased estimator on a
budget is worse than a noisy one, because you can carry a noisy estimate with a
margin and you cannot carry a systematically low one at all.

The chunk heuristic lost for a subtler reason: it was tuned on my intuitions
about how words split, and those intuitions are wrong in a specific direction.
Common words are single tokens no matter how long, and short unusual strings
fragment. Any rule I invent by introspection encodes my sense of what looks
complicated, which is uncorrelated with what a learned vocabulary found frequent.

Character count sidesteps all of it. It has no theory, so it has no wrong theory.

## More precision would have been fake

The obvious objection is: ship a real tokenizer and be exact. But exact against
*what*? Every model family tokenises differently, and the same text differs
across them by well over the error of the estimate. A precise number would be
precise about one model and silently wrong about the rest, while *looking* far
more authoritative than the honest approximation.

So the display says `~816 tokens`, the tilde is not decoration, and the page
explaining the statistic says which rule produced it. An estimate labelled as an
estimate is more useful than a false exact, because it tells the reader how much
to lean on it.

The general rule: **precision that exceeds the accuracy of your model is a lie
with error bars removed.** Round to the significant figures you can defend.

## The part that actually mattered: what to measure

This is where I nearly shipped something misleading, and it had nothing to do
with the estimator.

Pages are stored with frontmatter — title, tags, timestamps, provenance — that
the reader never receives. Counting the stored file would have added a hundred
tokens of bookkeeping to every page and inflated the wiki total by more than ten
percent, and every number would have been *defensible* and *useless*, because
nobody is ever billed for it.

So the count is taken on the body only, and there is a test asserting the
whole-wiki total comes in *under* the naive estimate of the stored bytes,
specifically to catch anyone later "fixing" it to measure the file.

The lesson generalises past tokens. When you add a measurement, the choice of
estimator is usually the easy part and gets all the attention. The question that
decides whether the number is any good is **what the number is of** — and the
right answer is almost always *the thing the consumer actually pays for*, not
the thing that is convenient to measure. Response bytes on the wire, not before
compression. Wall-clock the user waits, not CPU in the handler. Body tokens, not
file bytes.

## Two implementation notes worth stealing

**Store it, do not compute it.** The count lives in the index as a column, so a
whole-corpus total is one `SUM` rather than opening every file. Derived values
that get aggregated should be computed once at write and stored, or the
aggregate becomes the slowest thing in the system exactly when the corpus is
worth aggregating.

**Publish it before the fetch, not after.** The count appears in listings and
search results, not only on the page. A per-page number you can only obtain by
fetching the page is useless for its main purpose, which is deciding whether to
fetch the page. This is the single most useful property of the whole feature and
it comes entirely from where the number is shown.