1r"""
   2# Tokenization: how text becomes the integers a model actually reads
   3
   4Run: `python -m primer.ml.tokenization`
   5
   6New to the notation (sums, subscripts)? Every symbol is decoded where it
   7appears, and `primer.notation` teaches them all from zero. This lesson
   8builds on the embedding table of `primer.ml.big_picture`.
   9
  10## Level 1: The practitioner's guide
  11
  12**In one sentence.** A tokenizer cuts text into pieces from a fixed
  13vocabulary and hands the model a number for each piece; every price, rate
  14limit, context window and latency figure you meet is counted in those
  15pieces, and where their boundaries fall explains a whole family of odd model
  16behaviour.
  17
  18**When you need it.** On every call, because you are billed per token, but
  19you feel it on particular days: when you estimate a bill, when a prompt
  20overflows the context window, when users who write in Hindi or Japanese
  21cost several times what English users cost, when a model miscounts the
  22letters in a word or mis-adds two numbers, when a fine-tuning dataset is
  23quoted in tokens, or when you switch model generations and the same prompt
  24suddenly counts differently. The tell: a number that should be simple (how
  25long is this text?) that you cannot answer without running the model's own
  26tokenizer. This lesson's toy tokenizer, trained on English, spends 1.73
  27characters per token on an English sentence and 0.38 on the same sentence in
  28Hindi, 4.5 times as many tokens for the same meaning. You never train a
  29tokenizer unless you pretrain a model; you only ever count with one, and
  30choose models partly by theirs.
  31
  32**Your options.** From the quickest estimate to the most committed:
  33
  34| Option | What it does | What it gives you | What it costs | Where it lives |
  35|---|---|---|---|---|
  36| A rule of thumb | About 4 characters or 0.75 words per token of English prose | An instant estimate: 1,000 tokens is about 750 words | Wrong by several-fold for code, JSON and non-English text | Your head, and this lesson's `estimate_tokens` |
  37| Counting with the model's own tokenizer | Runs the real tokenizer offline (tiktoken, Hugging Face tokenizers, SentencePiece) | Exact counts for any model whose tokenizer ships with its weights | The right tokenizer files per model family; counts do not transfer between families | Your code |
  38| Counting through the API | A vendor endpoint counts a whole request: system prompt, tools, images, PDFs | Counts for models whose tokenizer is not published; free on Anthropic's API, with its own rate limit | One request per count; a close estimate rather than the exact bill | The API |
  39| Reading usage after each call | Every response reports the input and output tokens it consumed | The exact billed numbers, and the cache reads | Nothing beyond logging it | The API response |
  40| Shaping the text | Prefers prose to nested JSON, trims whitespace and decimals, shortens identifiers in tool outputs | Fewer tokens per call at the same meaning | Engineering time, and a risk of hurting clarity | Your prompt and your tool outputs |
  41| Choosing the model by its tokenizer | Picks a family whose vocabulary covers your languages | Shorter sequences and lower bills for non-English text | Larger vocabularies cost embedding parameters, paid by the vendor | The model card: `vocab_size` |
  42| Training or extending a vocabulary | Learns merges from your own corpus | The shortest sequences for your domain | The model must be trained or retrained with it | Your training stack |
  43
  44**How to choose.** Start from who runs the tokenizer.
  45
  46- A hosted model: count before you send when the size matters (context fit,
  47  routing between models) and read usage after every call for the bill.
  48  Recount when you change model generations: Anthropic's documentation says
  49  the tokenizer introduced with Claude Opus 4.7 produces about 30% more
  50  tokens for the same text than earlier models did.
  51- An open model: the tokenizer ships with the weights; load the pair
  52  together and never count with another family's tokenizer, because the
  53  pieces and the ids are unrelated.
  54- A multilingual product: budget per language with the real tokenizer,
  55  never with the English rule of thumb.
  56- A task about letters or digits (spelling, counting characters, arithmetic):
  57  give the model a tool or spell the item out, because the tokens hide the
  58  letters.
  59- A model with a chat template: apply it. Instruction-tuned models expect
  60  their special tokens in the right places, and a prompt built by hand
  61  produces ids the model was not trained on.
  62- Whatever you pick, measure with the tokenizer of the model you will
  63  actually call. The rule of thumb is for napkins.
  64
  65**What it costs.** Tokens are the unit of four bills.
  66
  67- Money. Two meters: this lesson's example call, 2,000 input tokens and a
  68  500-token answer at \$3 and \$15 per million, costs \$0.0135, and a million
  69  such calls cost \$13,500 (`estimate_cost`). Hosted APIs price output
  70  tokens several times higher than input tokens (5× across Anthropic's
  71  price list), so a verbose answer costs more than a long prompt.
  72- Latency. Each output token is one full step of generation, so answer
  73  length sets the wait; `primer.ml.inference` puts the step at about 4.78 ms
  74  for an 8-billion-parameter model on one GPU.
  75- Context. The window is counted in tokens, so the 4.5× gap between English
  76  and Hindi in this lesson's run is also a 4.5× gap in how much text fits.
  77- Vocabulary. A bigger kit means shorter sequences but a larger embedding
  78  table: GPT-2's 50,257 rows of width 768 are 38.6 million parameters, 32%
  79  of its smallest model (`primer.ml.transformer`). Llama 3 grew to a 128K
  80  vocabulary, 100K from the tiktoken kit plus 28K for other languages, and
  81  its paper reports better compression as a result.
  82
  83**What breaks.**
  84
  85- **Letters the model never saw.** " strawberry" is two ids in this lesson's
  86  tokenizer, ` straw` and `berry`, not ten letters. Counting the r's means
  87  recalling a spelling, not reading one.
  88- **Digits that split by length.** " 1234" is one token and " 12345" is two
  89  (" 1234", "5"), and "1234" at the start of a line is three, so place
  90  values never line up. Some newer tokenizers split digits into groups of at
  91  most three to soften this.
  92- **A leading space changes the token.** " cat" is id 303 and "cat" is ids
  93  99 and 268; the model must learn that both mean cat. A prompt that ends
  94  mid-word, or a prefix that gains a trailing space, changes the ids and
  95  breaks a cached prefix.
  96- **The same meaning at several times the price.** Hindi in this lesson
  97  costs 4.5× English; Petrov et al. (2023) measured differences of up to 15
  98  times between languages on production tokenizers. Budget per language.
  99- **The wrong tokenizer.** Fine-tuning data tokenized with another family's
 100  kit trains the model on unrelated ids. Load tokenizer and weights from the
 101  same checkpoint.
 102- **Estimates carried across generations.** A count measured on one model
 103  can be 30% short on its successor. Recount with the model id you will run.
 104
 105**In the wild.** GPT-2 introduced byte-level BPE with a pre-tokenization
 106regex, and most language-model tokenizers still follow that design. OpenAI's
 107tiktoken ships the `cl100k_base` and `o200k_base` vocabularies and claims to
 108be 3 to 6 times faster than a comparable open-source tokenizer; Hugging Face
 109tokenizers and Google's SentencePiece cover the open models, SentencePiece
 110writing spaces as the visible symbol `▁`. BERT uses WordPiece and T5 uses
 111Unigram. Llama 3's vocabulary is 128K entries, and Mistral's byte-fallback
 112tokenizer guarantees that no character ever becomes an unknown token.
 113Anthropic's `count_tokens` endpoint counts a full request (system prompt,
 114tools, images and PDFs) free of charge, subject to a rate limit of 5,000 to
 11520,000 requests per minute by usage tier, and every Messages API response
 116reports its `usage`. The papers are linked at the end of the lesson.
 117
 118**Go deeper.** Level 2 trains a tokenizer by hand on "low lower lowest" in
 119three merges, replays the merges to encode a word it never saw, drops to
 120bytes so that nothing is ever unknown, meets WordPiece and Unigram, and
 121prints the exact pieces behind each quirk above. If you only needed to
 122count and budget, you are done.
 123
 124## Level 2: How it works, from scratch
 125
 126Level 2 builds the kit of bricks from nothing, starting with what a token
 127is.
 128
 129## 1. Tokens: building words from a fixed kit of bricks
 130
 131**Everyday picture.** A box of Lego with a fixed set of bricks. Common
 132shapes have a ready-made brick; anything unusual gets assembled from smaller
 133bricks. You can build *anything*, but some builds take many more bricks than
 134others. A **tokenizer** is that kit for text: a fixed vocabulary of pieces
 135(typically 32,000 to 200,000 of them), each with a number, its **token id**.
 136
 137**Tiny worked example.** With a kit that contains `the`, `un`, `believ` and
 138`able`, "the" takes 1 brick and "unbelievable" takes 3: `un` + `believ` +
 139`able`. The model never sees the letters; it sees a short list of ids such
 140as `[262, 403, 11009, 540]`.
 141
 142| Approach | Kit size | "unbelievable" | Problem |
 143|---|---|---|---|
 144| Whole words | huge (every form of every word, every name) | 1 piece, if it's in the kit | words not in the kit become `<unk>`; typos break |
 145| Single bytes | 256 | 12 pieces | sequences 4-5× longer, and attention cost grows with length squared |
 146| **Subwords** | 32k to 200k | ~3 pieces | the middle ground every modern LLM uses |
 147
 148```mermaid
 149flowchart LR
 150  T["'the unbelievable'"] --> TOK["tokenizer<br/>(fixed kit of pieces)"]
 151  TOK --> P["pieces: 'the' | ' un' | 'believ' | 'able'"]
 152  P --> IDS["ids: 262, 403, 11009, 540"]
 153  IDS --> EMB["embedding table<br/>one vector per id"]
 154  EMB --> M["the model"]
 155```
 156
 157**Reading it:** text enters on the left and never reaches the model as
 158text. The tokenizer cuts it into pieces from its fixed kit and swaps each
 159piece for its number. Those numbers pick rows from the embedding table (see
 160`primer.ml.big_picture`), and only those vectors reach the model. Everything
 161the model knows about spelling it had to learn through this narrow window.
 162(The ids shown are illustrative.)
 163
 164**In code:** `trained_tokenizer` builds this lesson's toy kit, and
 165`ByteBPE.encode` turns any text into its list of ids.
 166
 167**Why it matters.** Everything downstream is counted in tokens: price,
 168speed, and how much fits in the context window.
 169
 170## 2. Byte Pair Encoding (BPE): learning the kit from data
 171
 172**Everyday picture.** A court stenographer inventing shorthand. Whatever
 173pair of letters they write most often gets its own squiggle. Then the most
 174common pair *including* that squiggle gets one too, and so on, until they
 175have as many squiggles as they can remember.
 176
 177**Tiny worked example.** Training text "low lower lowest". Start from single
 178letters and repeatedly merge the most frequent neighbouring pair
 179(`train_char_bpe` reproduces this exactly):
 180
 181| Step  | Most frequent pair | New token | Text becomes                    |
 182|-------|--------------------|-----------|---------------------------------|
 183| Start | none               | none      | l o w · l o w e r · l o w e s t |
 184| 1     | l + o (3 times)    | lo        | lo w · lo w e r · lo w e s t    |
 185| 2     | lo + w (3 times)   | low       | low · low e r · low e s t       |
 186| 3     | low + e (2 times)  | lowe      | low · lowe r · lowe s t         |
 187
 188At step 1, `l+o` and `o+w` are tied at 3; ties go to the pair seen first.
 189
 190```mermaid
 191flowchart LR
 192  T[Training text] --> S[Split into characters]
 193  S --> C[Count adjacent pairs]
 194  C --> M[Merge the most frequent pair<br/>record the merge rule]
 195  M -->|vocab not full yet| C
 196  M -->|vocab full| V[Vocabulary + ordered merge list]
 197```
 198
 199**Reading it:** training is a loop around two boxes. "Count adjacent pairs"
 200looks at every neighbouring pair of symbols in the corpus; "Merge" glues the
 201winner into one new symbol everywhere and appends that rule to an ordered
 202list. The loop exits when the vocabulary reaches its target size. The output
 203is not just a vocabulary but the *ordered* merge list, and encoding depends
 204on that order.
 205
 206**The math and the code.** Each round picks the pair with the largest count:
 207
 208$$
 209\text{count}(a, b) = \sum_{w} f_w \cdot n_w(a, b)
 210$$
 211
 212**Symbols**
 213
 214| Symbol | Meaning here | In the example |
 215|---|---|---|
 216| $a, b$ | two symbols that sit side by side | l and o |
 217| $w$ | one distinct word of the training text | "lower" |
 218| $\sum_{w}$ | "add up the following over every distinct word" | low, lower, lowest |
 219| $f_w$ | how many times word $w$ occurs in the text | 1 each |
 220| $n_w(a, b)$ | how many times $a$ is immediately followed by $b$ inside $w$ | 1 in each word |
 221| $\text{count}(a, b)$ | the pair's total, the number BPE ranks by | 3 |
 222
 223**In words:** "for each distinct word, count how often the pair appears
 224inside it, multiply by how often the word occurs, and add everything up."
 225
 226**With the numbers:** count(l, o) = 1·1 (low) + 1·1 (lower) + 1·1 (lowest) =
 227**3**; count(e, r) = 1·1 (lower) = **1**. `train_char_bpe` computes the same
 228totals with a `Counter`.
 229
 230**In Python:**
 231
 232```python
 233# f_w: how often each distinct word occurs
 234f = {"low": 1, "lower": 1, "lowest": 1}
 235# n_w(a, b): a immediately followed by b inside w
 236def n(w, a, b):
 237    return sum(1 for i in range(len(w) - 1) if w[i] == a and w[i + 1] == b)
 238def count(a, b):
 239    # Σ_w f_w · n_w(a, b)
 240    return sum(f_w * n(w, a, b) for w, f_w in f.items())
 241count("l", "o"), count("e", "r")  # → (3, 1)
 242```
 243
 244**In code:** `train_char_bpe` runs the count-and-merge loop and returns one
 245`MergeStep` per round, holding the winning pair, its count, the new token
 246and the text after the merge (one row of the table above).
 247
 248**Why it matters.** Frequent strings end up as single tokens and rare ones
 249stay in pieces, which is exactly why token counts differ from word counts,
 250and why a tokenizer trained mostly on English is cheap for English and
 251expensive for everything else (section 6).
 252
 253## 3. Encoding new text: replay the merges in order
 254
 255**Everyday picture.** Following a recipe card: do step 1 everywhere it
 256applies, then step 2, and so on. Skipping ahead would give a different dish.
 257
 258**Tiny worked example.** Encode "lowest" with the three merges learned
 259above: `l o w e s t` → (merge 1) `lo w e s t` → (merge 2) `low e s t` →
 260(merge 3) `lowe s t`. Result: 3 tokens. A word never seen in training still
 261works: "slow" → `s low`; "newer" matches no merge and stays as letters.
 262
 263```mermaid
 264flowchart LR
 265  W["'lowest' as letters<br/>l o w e s t"] --> R1["merge 1: l+o<br/>lo w e s t"]
 266  R1 --> R2["merge 2: lo+w<br/>low e s t"]
 267  R2 --> R3["merge 3: low+e<br/>lowe s t"]
 268  R3 --> OUT["no learned merge applies<br/>tokens: lowe | s | t"]
 269```
 270
 271**Reading it:** each box applies one merge rule, in the order the rules
 272were learned, to every place it fits. The word shrinks from 6 symbols to 3.
 273It stops when none of the remaining neighbouring pairs is in the rule list.
 274
 275**The code.** `encode_char_bpe` repeatedly finds, among the pairs present,
 276the one learned *earliest*, merges it, and loops. Training and encoding
 277agree because they apply rules in the same order.
 278
 279**Why it matters.** Encoding is deterministic: the same text always gives
 280the same ids, which is what makes prompt caching and cost estimates possible.
 281
 282## 4. Byte-level BPE: nothing is ever unknown
 283
 284**Everyday picture.** Every file on a computer, whatever it holds, is a
 285sequence of **bytes**: numbers from 0 to 255. If the smallest bricks in the
 286kit are those 256 byte values, then no text can ever be unbuildable: emoji,
 287Chinese, source code or garbage all come apart into bytes.
 288
 289**Tiny worked example.** In UTF-8 (the standard way to store text as bytes)
 290"a" is one byte, 97; "é" is two bytes, 195 169; "🙂" is four bytes,
 291240 159 153 130. An untrained byte-level tokenizer turns "é🙂" into 6 tokens.
 292After training on English, " password" is one token, id 291, because it was
 293frequent.
 294
 295```mermaid
 296flowchart LR
 297  T["Text: 'Reset your password'"] --> P["Pre-tokenize (regex)<br/>'Reset' | ' your' | ' password'"]
 298  P --> B["UTF-8 bytes per chunk<br/>' your' = 32 121 111 117 114"]
 299  B --> R["Replay merges, earliest first<br/>inside each chunk only"]
 300  R --> I["Token ids<br/>346 377 309 328 291"]
 301  I --> D["Decode: look up bytes per id,<br/>concatenate, UTF-8 decode"]
 302```
 303
 304**Reading it:** encoding runs left to right. A **regular expression** (a
 305text-matching pattern) first cuts the text into chunks, each keeping its
 306leading space; merges never cross a chunk boundary. Each chunk becomes raw
 307bytes (ids 0-255). The tokenizer then applies whichever learned merge came
 308earliest among the pairs present, until none applies. The ids shown are what
 309this module's toy tokenizer produces: "Reset" is two tokens (`Re` `set`),
 310" password" is one. Decoding is the reverse lookup: every id maps to a fixed
 311byte string, and concatenating them restores the exact original bytes. That
 312is why a byte-level tokenizer round-trips any text.
 313
 314**The code.** `ByteBPE` is `train_char_bpe` with bytes instead of letters,
 315ids 0-255 for the bytes and 256, 257, ... for each merge. GPT-2, GPT-4's
 316`cl100k_base`, Llama 3 and most current LLMs work this way.
 317
 318![Compression climbs steeply from 1.0 to 2.0 characters per token in the first 44 merges, then flattens near 3.1 by 500 entries](figures/primer.ml.tokenization.merge_curve.svg)
 319
 320**Reading it:** the x-axis is vocabulary size: 256 raw bytes plus the
 321number of merges learned. The y-axis is compression, how many characters of
 322held-out English each token covers on average. At 256 (no merges) every byte
 323is a token, so the value is 1. The first few dozen merges capture the most
 324frequent pairs (" t", "he", " the") and the curve climbs steeply; later
 325merges capture rarer strings and the curve flattens. (Our toy corpus runs
 326out of repeated strings near 500; real corpora keep going.) Production
 327tokenizers sit far to the right, at 50k to 200k entries, and reach about 4
 328characters per token on English. Bigger kits mean shorter sequences but a
 329larger embedding table and output layer.
 330
 331**In code:** `ByteBPE.train` learns the byte merges, `ByteBPE.encode` and
 332`ByteBPE.decode` make the round trip, and `compression_curve` trains
 333tokenizers of growing size and measures each one for the figure above.
 334
 335**Try it:** the tokenizer below is this lesson's own, with its 144 learned
 336merges. Drag the merges down to 0 and every byte is its own token; drag them
 337back up and watch frequent pieces such as " the" and " password" fuse, one
 338merge at a time, while characters per token climbs like the curve above. Then
 339type a word the corpus never saw, or an accent or an emoji, and watch it stay
 340in small pieces.
 341
 342<div class="viz" data-viz="tokenizer" aria-label="Byte-level BPE tokenizer: type text and choose how many merges apply"></div>
 343
 344**Why it matters.** No "unknown token" failures, ever, for any input. The
 345price is that unfamiliar scripts fall back to near-byte level and cost many
 346more tokens.
 347
 348## 5. The cousins: WordPiece and Unigram
 349
 350**Everyday picture.** BPE glues the *most common* pair. WordPiece glues the
 351most *inseparable* pair: two pieces that almost never appear apart, like
 352"Q" and "u" in English. Unigram works the other way round: start with a huge
 353kit and keep throwing out the bricks you'd miss least.
 354
 355**Tiny worked example.** On "low lower lowest", BPE's first merge is `l+o`.
 356WordPiece's first merge is `s+t`: "s" and "t" each appear exactly once, and
 357always together.
 358
 359```mermaid
 360flowchart TB
 361  subgraph BPE["BPE (GPT, Llama)"]
 362    b1[start small] --> b2[merge most frequent pair] --> b3[grow to target size]
 363  end
 364  subgraph WP["WordPiece (BERT)"]
 365    w1[start small] --> w2["merge pair with best<br/>count(ab) / count(a)·count(b)"] --> w3[grow to target size]
 366  end
 367  subgraph UG["Unigram (T5, SentencePiece default)"]
 368    u1[start huge] --> u2[drop pieces whose loss<br/>hurts likelihood least] --> u3[shrink to target size]
 369  end
 370```
 371
 372**Reading it:** the two top rows grow a vocabulary bottom-up and differ only
 373in how they rank candidate pairs. The bottom row prunes top-down. All three
 374end with a fixed kit and a rule for cutting new text into pieces from it.
 375
 376**The math.** WordPiece ranks pairs by
 377
 378$$
 379\text{score}(a, b) = \frac{\text{count}(ab)}{\text{count}(a) \cdot \text{count}(b)}
 380$$
 381
 382**Symbols**
 383
 384| Symbol | Meaning here | In the example |
 385|---|---|---|
 386| $\text{count}(ab)$ | how often $a$ is immediately followed by $b$ | count(st) = 1 |
 387| $\text{count}(a)$ | how often $a$ appears at all | count(s) = 1 |
 388| $\text{count}(b)$ | how often $b$ appears at all | count(t) = 1 |
 389| fraction bar | divide: pairs are rewarded when their parts rarely appear apart | |
 390
 391**In words:** "how often the two appear together, divided by how often each
 392appears at all."
 393
 394**With the numbers:** score(s, t) = 1 / (1 · 1) = **1.0**; score(l, o) =
 3953 / (3 · 3) = **0.33**. WordPiece merges s+t first (`wordpiece_scores`).
 396
 397**In Python:**
 398
 399```python
 400f = {"low": 1, "lower": 1, "lowest": 1}
 401# how often a symbol, or a pair written together, appears
 402def count(piece):
 403    return sum(f_w * w.count(piece) for w, f_w in f.items())
 404def score(a, b):
 405    # count(ab) / (count(a) · count(b))
 406    return count(a + b) / (count(a) * count(b))
 407score("s", "t"), round(score("l", "o"), 2)  # → (1.0, 0.33)
 408```
 409
 410BERT marks continuation pieces with `##` ("un", "##believ", "##able").
 411**SentencePiece** is a library that runs BPE or Unigram directly on raw text,
 412writing spaces as the visible symbol `▁`.
 413
 414**Why it matters.** Different model families segment the same text
 415differently, so their token counts and costs differ. Always count with the
 416tokenizer of the model you'll actually call.
 417
 418## 6. Tokens are the unit of cost, speed and memory
 419
 420**Everyday picture.** A taxi meter that ticks per token, not per mile, with
 421a pricier meter for the return trip: output tokens usually cost several
 422times more than input tokens.
 423
 424**Tiny worked example.** A prompt of 2,000 tokens with a 500-token answer, at
 425example prices of \$3 per million input tokens and \$15 per million output
 426tokens: 0.006 + 0.0075 = **\$0.0135**. A million such calls cost \$13,500.
 427
 428```mermaid
 429flowchart LR
 430  P["prompt text"] --> TI["count input tokens<br/>meter A: $ per million in"]
 431  TI --> M[model]
 432  M --> TO["count output tokens<br/>meter B: $ per million out (pricier)"]
 433  TO --> BILL["bill = A + B"]
 434  TI -.-> CTX["also: must fit the<br/>context window"]
 435```
 436
 437**Reading it:** two meters run on every call. Input tokens are counted once
 438on the way in; they also have to fit in the model's context window (the
 439dotted arrow). Output tokens are counted on the way out, and each one also
 440takes a full step of generation, so they drive latency as well as cost.
 441
 442**The math and the code.**
 443
 444$$
 445\text{cost} = \frac{T_{in}}{10^6}\,p_{in} + \frac{T_{out}}{10^6}\,p_{out}
 446$$
 447
 448**Symbols**
 449
 450| Symbol | Meaning here | In the example |
 451|---|---|---|
 452| $T_{in}$ | input tokens in the call | 2,000 |
 453| $T_{out}$ | output tokens generated | 500 |
 454| $10^6$ | one million, because prices are quoted per million tokens | 1,000,000 |
 455| $p_{in}$ | price per million input tokens, in dollars | 3 |
 456| $p_{out}$ | price per million output tokens, in dollars | 15 |
 457
 458**In words:** "input tokens in millions times the input price, plus output
 459tokens in millions times the output price."
 460
 461**With the numbers:** 2,000/1,000,000 × 3 + 500/1,000,000 × 15 = 0.006 +
 4620.0075 = **\$0.0135** (`estimate_cost`).
 463
 464**In Python:**
 465
 466```python
 467T_in, T_out = 2_000, 500
 468# dollars per million tokens
 469p_in, p_out = 3, 15
 470round(T_in / 10**6 * p_in + T_out / 10**6 * p_out, 4)  # → 0.0135
 471```
 472
 473**Rule of thumb:** English prose averages about **4 characters per token**,
 474or **0.75 words per token**, so 1,000 tokens ≈ 750 words
 475(`estimate_tokens`, `tokens_to_words`). Use it for quick estimates; count
 476with the real tokenizer for anything that matters.
 477
 478![English prose gets 2.6 characters per token; German, code and JSON 1.2 to 1.4; Hindi and emoji under 0.4](figures/primer.ml.tokenization.content_types.svg)
 479
 480**Reading it:** each bar is one kind of text encoded by the same
 481English-trained tokenizer from this module; taller bars mean each token
 482carries more text, so the content is cheaper. English prose scores highest
 483because the merges were learned from English. Code and JSON score lower
 484because symbols and unusual identifiers were rarely merged. German shares the
 485alphabet but not the words. Hindi and emoji fall below one character per
 486token: each character is 3-4 UTF-8 bytes and few of those byte pairs were
 487ever merged, the worst case of byte-level fallback.
 488
 489**In code:** `chars_per_token` computes each bar: the length of the text
 490divided by the number of ids `ByteBPE.encode` returns for it.
 491
 492**Why it matters.** The same request can differ several-fold in cost,
 493latency and context usage depending on language and content: dense JSON,
 494code, tables and non-English text all need more tokens than English prose.
 495
 496## 7. Quirks the tokenizer explains
 497
 498**Everyday picture.** Reading through frosted glass that only shows whole
 499bricks. You can tell which bricks are there, but not the letters printed on
 500them.
 501
 502**Tiny worked example** (this module's toy tokenizer):
 503
 504| Text | Tokens | What it explains |
 505|---|---|---|
 506| `" cat"` vs `"cat"` | `[303]` vs `[99, 268]` (`c` `at`) | a leading space makes a different token; the model must learn they mean the same |
 507| `" 1234"` vs `" 12345"` | `[' 1234']` vs `[' 1234', '5']` | digits split by length and context, so place values don't line up: one reason arithmetic is hard |
 508| `" strawberry"` | `[' straw', 'berry']` | 2 ids, not 10 letters: "how many r's?" is hard |
 509
 510```mermaid
 511flowchart LR
 512  W["' strawberry'"] --> T["tokenizer"] --> I["ids 396, 311"]
 513  I --> M["model sees two ids<br/>no letters, no count of r"]
 514  Q["'how many r's?'"] --> M
 515```
 516
 517**Reading it:** the question is about letters, but the letters were
 518discarded before the model saw anything. It has to have *memorized* how
 519` straw` and `berry` are spelled from seeing them in text, which is why
 520spelling and letter-counting questions trip up models that handle much
 521harder reasoning.
 522
 523**The code.** `ByteBPE.tokens(text)` shows the pieces for any string; the
 524demo prints these cases.
 525
 526**Why it matters.** These quirks explain surprising behaviour: miscounted
 527letters, shaky arithmetic, odd handling of rare names. Some newer tokenizers
 528split digits into fixed groups of up to 3 to reduce the arithmetic problem.
 529
 530## In 20 seconds
 531- Text is split into subword pieces by BPE: start from characters or bytes
 532  and repeatedly merge the most frequent neighbouring pair.
 533- Byte-level BPE can encode anything with no unknown token.
 534- Cost, latency and context limits are counted in tokens; English is about
 535  4 characters (0.75 words) per token, while code and non-English text need
 536  more tokens.
 537
 538## Self-test questions
 539
 540**Why subwords instead of whole words or characters?**
 541Words give a huge vocabulary and unknown words; characters make sequences
 542several times longer, and attention cost grows with the square of length.
 543Subwords keep common strings short and still encode anything.
 544
 545**How does BPE training proceed, step by step, on "low lower lowest"?**
 546Split into letters, count neighbouring pairs, and merge the most frequent:
 547l+o (3), then lo+w (3), then low+e (2). Record each merge; to encode, replay
 548the merges in the order learned.
 549
 550**Why can a byte-level BPE tokenizer never produce an unknown token?**
 551Its base vocabulary is all 256 byte values, and every string is a sequence of
 552UTF-8 bytes, so in the worst case text falls back to single bytes.
 553
 554**What does WordPiece do differently from BPE?**
 555It merges the pair with the highest count(ab) / (count(a)·count(b)), which
 556favours pairs whose parts rarely appear apart, instead of the most frequent
 557pair.
 558
 559**Your users write in Hindi. What changes in your estimates?**
 560Expect noticeably more tokens for the same content: higher cost and latency,
 561and less text per context window. Measure with the actual tokenizer.
 562
 563**Why are LLMs bad at counting letters and at long arithmetic?**
 564They see token ids, not characters, and numbers split into chunks that don't
 565line up with place value.
 566
 567**What does a call with 2,000 input and 500 output tokens cost at \$3/\$15 per million?**
 5680.006 + 0.0075 = \$0.0135. And 1,000 tokens is about 750 English words.
 569
 570## The papers behind this lesson
 571
 572- **Sennrich, Haddow & Birch (2015), *Neural Machine Translation of Rare Words with Subword Units*.**
 573  https://arxiv.org/abs/1508.07909. Brought byte pair encoding, a 1990s
 574  compression trick, to language models as a way to build open-vocabulary
 575  subword units. [annotated companion](../../papers/bpe-subwords.html)
 576- **Radford et al. (2019), *Language Models are Unsupervised Multitask Learners* (GPT-2).**
 577  https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf.
 578  Introduced byte-level BPE with a pre-tokenization regex, the design most
 579  LLM tokenizers still follow.
 580- **Kudo & Richardson (2018), *SentencePiece*.** https://arxiv.org/abs/1808.06226.
 581  A language-independent tokenizer library that works on raw text and
 582  encodes spaces as the symbol ▁.
 583- **Kudo (2018), *Subword Regularization*.** https://arxiv.org/abs/1804.10959.
 584  Introduced the Unigram language-model tokenizer that prunes a large
 585  vocabulary down instead of merging up.
 586
 587## Further reading
 588- Sennrich et al., *Neural Machine Translation of Rare Words with Subword Units* (the BPE paper, 2015): https://arxiv.org/abs/1508.07909
 589- Andrej Karpathy, *Let's build the GPT Tokenizer* (video): https://www.youtube.com/watch?v=zduSFxRajkE
 590- Karpathy's `minbpe` (minimal, clean byte-level BPE): https://github.com/karpathy/minbpe
 591- OpenAI `tiktoken` (fast BPE used by GPT models): https://github.com/openai/tiktoken
 592- Hugging Face LLM course, BPE chapter: https://huggingface.co/learn/llm-course/chapter6/5
 593- Hugging Face LLM course, WordPiece chapter: https://huggingface.co/learn/llm-course/chapter6/6
 594- Hugging Face LLM course, Unigram chapter: https://huggingface.co/learn/llm-course/chapter6/7
 595- Kudo & Richardson, *SentencePiece* (2018): https://arxiv.org/abs/1808.06226
 596- Kudo, *Subword Regularization* (the Unigram LM, 2018): https://arxiv.org/abs/1804.10959
 597- Petrov et al., *Language Model Tokenizers Introduce Unfairness Between Languages* (2023): https://arxiv.org/abs/2305.15425
 598"""
 599
 600from __future__ import annotations
 601
 602import math
 603import re
 604from collections import Counter
 605from dataclasses import dataclass
 606
 607from primer._show import banner, say, table, takeaway
 608
 609# ---------------------------------------------------------------------------
 610# 1. Character-level BPE: the textbook algorithm, reproducing the worked example
 611# ---------------------------------------------------------------------------
 612
 613
 614@dataclass
 615class MergeStep:
 616    """One BPE training step: which pair merged, how often it occurred, and the result."""
 617
 618    pair: tuple[str, str]
 619    count: int
 620    new_token: str
 621    text_after: str
 622
 623
 624def _count_pairs(words: dict[tuple[str, ...], int]) -> tuple[Counter, dict[tuple[str, str], int]]:
 625    """Count adjacent symbol pairs, weighted by word frequency.
 626
 627    Also records the order in which each pair was *first seen*, used to break
 628    ties deterministically (earliest pair wins).
 629    """
 630    counts: Counter = Counter()
 631    first_seen: dict[tuple[str, str], int] = {}
 632    for symbols, freq in words.items():  # dicts preserve insertion (= corpus) order
 633        for pair in zip(symbols, symbols[1:]):
 634            counts[pair] += freq
 635            first_seen.setdefault(pair, len(first_seen))
 636    return counts, first_seen
 637
 638
 639def _merge_symbols(symbols: tuple[str, ...], pair: tuple[str, str]) -> tuple[str, ...]:
 640    """Replace every occurrence of `pair` in `symbols` with the concatenated symbol."""
 641    out, i = [], 0
 642    while i < len(symbols):
 643        if i + 1 < len(symbols) and (symbols[i], symbols[i + 1]) == pair:
 644            out.append(symbols[i] + symbols[i + 1])
 645            i += 2
 646        else:
 647            out.append(symbols[i])
 648            i += 1
 649    return tuple(out)
 650
 651
 652def _render(occurrences: list[str], words: dict[str, tuple[str, ...]]) -> str:
 653    """Show the corpus as symbols, `·` between words, like the worked-example table."""
 654    return " · ".join(" ".join(words[w]) for w in occurrences)
 655
 656
 657def train_char_bpe(text: str, num_merges: int) -> list[MergeStep]:
 658    """Learn `num_merges` BPE merges over the characters of whitespace-split words.
 659
 660    >>> [s.new_token for s in train_char_bpe("low lower lowest", 3)]
 661    ['lo', 'low', 'lowe']
 662    """
 663    occurrences = text.split()
 664    # Each distinct word -> its current symbol sequence; frequencies -> weights.
 665    freq = Counter(occurrences)
 666    current: dict[str, tuple[str, ...]] = {w: tuple(w) for w in dict.fromkeys(occurrences)}
 667    steps: list[MergeStep] = []
 668    for _ in range(num_merges):
 669        counts, first_seen = _count_pairs({current[w]: freq[w] for w in current})
 670        if not counts:
 671            break  # every word is a single symbol; nothing left to merge
 672        # Most frequent pair; ties broken by earliest first occurrence.
 673        pair = max(counts, key=lambda p: (counts[p], -first_seen[p]))
 674        for w in current:
 675            current[w] = _merge_symbols(current[w], pair)
 676        steps.append(MergeStep(pair, counts[pair], pair[0] + pair[1], _render(occurrences, current)))
 677    return steps
 678
 679
 680def encode_char_bpe(word: str, merges: list[tuple[str, str]]) -> list[str]:
 681    """Tokenize one word by replaying learned merges in the order they were learned.
 682
 683    Replaying by *rank* (earliest-learned merge first) is what makes encoding
 684    deterministic and consistent with training.
 685
 686    >>> encode_char_bpe("lowest", [("l", "o"), ("lo", "w"), ("low", "e")])
 687    ['lowe', 's', 't']
 688    """
 689    rank = {p: i for i, p in enumerate(merges)}
 690    symbols = tuple(word)
 691    while len(symbols) > 1:
 692        # Among pairs present, find the one learned earliest.
 693        candidates = [p for p in zip(symbols, symbols[1:]) if p in rank]
 694        if not candidates:
 695            break
 696        symbols = _merge_symbols(symbols, min(candidates, key=rank.__getitem__))
 697    return list(symbols)
 698
 699
 700def wordpiece_scores(text: str) -> dict[tuple[str, str], float]:
 701    """WordPiece's merge score for every adjacent pair: count(ab) / (count(a)·count(b)).
 702
 703    BPE picks the most *frequent* pair. WordPiece picks the pair whose parts
 704    are most strongly associated (they rarely occur apart), which is the
 705    merge that most increases likelihood under a unigram model.
 706    """
 707    occurrences = text.split()
 708    words = Counter(tuple(w) for w in occurrences)
 709    pair_counts, _ = _count_pairs(dict(words))
 710    sym_counts: Counter = Counter()
 711    for symbols, f in words.items():
 712        for s in symbols:
 713            sym_counts[s] += f
 714    return {p: c / (sym_counts[p[0]] * sym_counts[p[1]]) for p, c in pair_counts.items()}
 715
 716
 717# ---------------------------------------------------------------------------
 718# 2. Byte-level BPE: what GPT-2, GPT-4, Llama 3 etc. actually do
 719# ---------------------------------------------------------------------------
 720
 721# Pre-tokenization regex, a simplified GPT-2 pattern. It splits text into
 722# chunks BEFORE merging, so merges never cross these boundaries:
 723#   * English contractions ('s, 't, ...)
 724#   * an optional leading space + a run of letters     -> " hello"
 725#   * an optional leading space + a run of digits      -> " 1234"
 726#   * an optional leading space + a run of other stuff -> " {}", "!!"
 727#   * whitespace runs
 728# The "optional leading space" is why " cat" (mid-sentence) and "cat"
 729# (start of text) become different tokens.
 730PRETOKENIZE = re.compile(r"""'(?:s|t|re|ve|m|ll|d)| ?[A-Za-z]+| ?[0-9]+| ?[^\sA-Za-z0-9]+|\s+(?!\S)|\s+""")
 731
 732
 733class ByteBPE:
 734    """A minimal byte-level BPE tokenizer (train, encode, decode).
 735
 736    Token ids 0..255 are the raw bytes. Every learned merge adds one id:
 737    256, 257, ... `vocab[id]` holds the bytes each id stands for.
 738    """
 739
 740    def __init__(self) -> None:
 741        self.merges: dict[tuple[int, int], int] = {}  # (a, b) -> new id, in learned order
 742        self.vocab: dict[int, bytes] = {i: bytes([i]) for i in range(256)}
 743
 744    @property
 745    def vocab_size(self) -> int:
 746        return len(self.vocab)
 747
 748    def train(self, text: str, vocab_size: int) -> "ByteBPE":
 749        """Learn merges until the vocabulary has `vocab_size` entries (>= 256)."""
 750        assert vocab_size >= 256, "the 256 byte values are always in the vocabulary"
 751        # Distinct chunks with counts: merging per distinct chunk is much faster
 752        # than walking the whole text on every merge.
 753        chunks = Counter(tuple(c.encode("utf-8")) for c in PRETOKENIZE.findall(text))
 754        next_id = 256
 755        while next_id < vocab_size:
 756            pairs: Counter = Counter()
 757            for ids, f in chunks.items():
 758                for p in zip(ids, ids[1:]):
 759                    pairs[p] += f
 760            if not pairs:
 761                break
 762            # Most frequent; ties -> smallest pair (deterministic across runs).
 763            best = max(pairs, key=lambda p: (pairs[p], -p[0], -p[1]))
 764            self.merges[best] = next_id
 765            self.vocab[next_id] = self.vocab[best[0]] + self.vocab[best[1]]
 766            new_chunks: Counter = Counter()
 767            for ids, f in chunks.items():
 768                new_chunks[self._merge(ids, best, next_id)] += f
 769            chunks = new_chunks
 770            next_id += 1
 771        return self
 772
 773    @staticmethod
 774    def _merge(ids: tuple[int, ...], pair: tuple[int, int], new_id: int) -> tuple[int, ...]:
 775        out, i = [], 0
 776        while i < len(ids):
 777            if i + 1 < len(ids) and (ids[i], ids[i + 1]) == pair:
 778                out.append(new_id)
 779                i += 2
 780            else:
 781                out.append(ids[i])
 782                i += 1
 783        return tuple(out)
 784
 785    def _encode_chunk(self, chunk: str) -> list[int]:
 786        ids = tuple(chunk.encode("utf-8"))
 787        while len(ids) > 1:
 788            # Apply the earliest-learned merge available (lowest new id).
 789            present = [p for p in zip(ids, ids[1:]) if p in self.merges]
 790            if not present:
 791                break
 792            pair = min(present, key=self.merges.__getitem__)
 793            ids = self._merge(ids, pair, self.merges[pair])
 794        return list(ids)
 795
 796    def encode(self, text: str) -> list[int]:
 797        """Text -> token ids. Works for ANY string: worst case, one id per byte."""
 798        return [i for chunk in PRETOKENIZE.findall(text) for i in self._encode_chunk(chunk)]
 799
 800    def decode(self, ids: list[int]) -> str:
 801        """Token ids -> text. Invalid UTF-8 (e.g. half an emoji) is replaced, never crashes."""
 802        return b"".join(self.vocab[i] for i in ids).decode("utf-8", errors="replace")
 803
 804    def tokens(self, text: str) -> list[str]:
 805        """Human-readable pieces, for display. Partial multi-byte chars show as '\\x..'."""
 806        out = []
 807        for i in self.encode(text):
 808            b = self.vocab[i]
 809            try:
 810                out.append(b.decode("utf-8"))
 811            except UnicodeDecodeError:
 812                out.append("".join(f"\\x{x:02x}" for x in b))
 813        return out
 814
 815
 816# A small English training corpus. Repetition stands in for "web scale":
 817# frequent strings (" the", " password", " 1234") become single tokens.
 818TRAINING_TEXT = (
 819    """
 820The cat sat on the mat. The cat saw the other cat and the dog.
 821Reset your password in the portal. Your password must be long. The password
 822policy says the password expires every year. Error code 1234 means the network
 823is down; error code 1234 again means the network is still down. Call 1234.
 824The model reads tokens, not words. The tokenizer splits text into tokens and
 825the model predicts the next token. Tokens are counted for cost and for context.
 826We ate the strawberry and the blueberry. The berry was sweet. Straw hats and
 827straw men. The quick brown fox jumps over the lazy dog.
 828"""
 829    * 20
 830)
 831
 832
 833def trained_tokenizer(vocab_size: int = 400) -> ByteBPE:
 834    """The byte-level tokenizer used by the demos and by `primer.ml.big_picture`."""
 835    return ByteBPE().train(TRAINING_TEXT, vocab_size)
 836
 837
 838# ---------------------------------------------------------------------------
 839# 3. Rules of thumb for cost and context estimates
 840# ---------------------------------------------------------------------------
 841
 842CHARS_PER_TOKEN = 4.0  # English prose, typical modern tokenizer
 843WORDS_PER_TOKEN = 0.75
 844
 845
 846def estimate_tokens(text: str) -> int:
 847    """Quick estimate: ~4 characters per token for English prose."""
 848    return max(1, math.ceil(len(text) / CHARS_PER_TOKEN))
 849
 850
 851def tokens_to_words(tokens: int) -> float:
 852    """1,000 tokens ≈ 750 words."""
 853    return tokens * WORDS_PER_TOKEN
 854
 855
 856def estimate_cost(
 857    input_tokens: int, output_tokens: int, usd_per_m_input: float, usd_per_m_output: float
 858) -> float:
 859    """Dollar cost of one call, given per-million-token prices.
 860
 861    Output tokens are typically priced several times higher than input tokens,
 862    so long answers cost more than long prompts. Look up current prices on
 863    your provider's pricing page; they change often.
 864    """
 865    return input_tokens / 1e6 * usd_per_m_input + output_tokens / 1e6 * usd_per_m_output
 866
 867
 868def chars_per_token(tok: ByteBPE, text: str) -> float:
 869    return len(text) / len(tok.encode(text))
 870
 871
 872# ---------------------------------------------------------------------------
 873# 4. Figures (rendered to docs/figures by `make figures`)
 874# ---------------------------------------------------------------------------
 875
 876HELD_OUT_ENGLISH = (
 877    "The user asked the model to reset the password and explain the error code. "
 878    "The tokenizer counts every token, and the model predicts the next token from the tokens before it."
 879)
 880
 881CONTENT_SAMPLES = {
 882    "English prose": HELD_OUT_ENGLISH,
 883    "German": "Der Benutzer bat das Modell, das Passwort zurückzusetzen und den Fehlercode zu erklären.",
 884    "Python code": "def reset_password(user_id: int) -> bool:\n    return api.reset(user_id=user_id, force=True)",
 885    "JSON": '{"user_id": 81723, "action": "reset_password", "force": true, "ts": "2026-01-05T10:22:31Z"}',
 886    "Hindi": "उपयोगकर्ता ने मॉडल से पासवर्ड रीसेट करने और त्रुटि कोड समझाने को कहा।",
 887    "Emoji": "🙂🙃😀🚀🔥✅❌👍🎉💡",
 888}
 889
 890
 891def compression_curve(vocab_sizes=(256, 270, 300, 350, 400, 450, 500)) -> list[tuple[int, float]]:
 892    """(vocab size, characters per token on held-out English) for tokenizers of growing size."""
 893    return [(v, chars_per_token(ByteBPE().train(TRAINING_TEXT, v), HELD_OUT_ENGLISH)) for v in vocab_sizes]
 894
 895
 896def viz_data() -> dict:
 897    """What the site's interactive tokenizer starts from: this lesson's merges and a sentence."""
 898    tok = trained_tokenizer()
 899    return {
 900        "tokenizer": {
 901            # Byte pairs in learned order: merge i makes id 256 + i. The widget
 902            # rebuilds each token's bytes from these and replays them like
 903            # ByteBPE.encode, so the slider's n is "the first n merges".
 904            "merges": [list(pair) for pair in tok.merges],
 905            # The first held-out sentence: English the merges were not learned from.
 906            "example": HELD_OUT_ENGLISH[: HELD_OUT_ENGLISH.index(".") + 1],
 907        }
 908    }
 909
 910
 911def figures() -> dict:
 912    """Data figures for this lesson: {key: matplotlib Figure}."""
 913    import matplotlib
 914
 915    matplotlib.use("Agg")
 916    import matplotlib.pyplot as plt
 917
 918    figs = {}
 919
 920    curve = compression_curve()
 921    fig, ax = plt.subplots(figsize=(6.5, 4))
 922    ax.plot([v for v, _ in curve], [c for _, c in curve], marker="o")
 923    ax.set_xlabel("vocabulary size (256 bytes + learned merges)")
 924    ax.set_ylabel("characters per token (held-out English)")
 925    ax.set_title("More merges compress text, with diminishing returns")
 926    ax.grid(alpha=0.3)
 927    fig.tight_layout()
 928    figs["merge_curve"] = fig
 929
 930    tok = trained_tokenizer()
 931    names = list(CONTENT_SAMPLES)
 932    values = [chars_per_token(tok, CONTENT_SAMPLES[n]) for n in names]
 933    fig, ax = plt.subplots(figsize=(6.5, 4))
 934    ax.bar(names, values)
 935    ax.set_ylabel("characters per token (higher = cheaper)")
 936    ax.set_title("Same tokenizer, very different cost by content")
 937    ax.tick_params(axis="x", rotation=25)
 938    ax.grid(axis="y", alpha=0.3)
 939    fig.tight_layout()
 940    figs["content_types"] = fig
 941    return figs
 942
 943
 944# ---------------------------------------------------------------------------
 945# 5. Narrated walkthrough
 946# ---------------------------------------------------------------------------
 947
 948
 949def demo() -> None:
 950    banner("1. BPE by hand: 'low lower lowest' in three merges")
 951    steps = train_char_bpe("low lower lowest", 3)
 952    rows = [("start", "none", "none", "l o w · l o w e r · l o w e s t")]
 953    rows += [(i + 1, f"{s.pair[0]} + {s.pair[1]} ({s.count} times)", s.new_token, s.text_after) for i, s in enumerate(steps)]
 954    table(["step", "most frequent pair", "new token", "text becomes"], rows)
 955    merges = [s.pair for s in steps]
 956    say(
 957        f"""
 958        Encoding replays the merges in learned order. A word never seen in
 959        training still works: 'lowest' -> {encode_char_bpe('lowest', merges)},
 960        'slow' -> {encode_char_bpe('slow', merges)}, 'newer' ->
 961        {encode_char_bpe('newer', merges)} (no merges apply, so characters).
 962        """
 963    )
 964    ws = wordpiece_scores("low lower lowest")
 965    best = max(ws, key=ws.get)
 966    say(
 967        f"""
 968        WordPiece scores pairs by count(ab)/(count(a)·count(b)) instead. Its first
 969        merge would be {best[0]}+{best[1]} (score {ws[best]:.2f}), not l+o
 970        (score {ws[('l', 'o')]:.2f}): 's' and 't' never appear apart.
 971        """
 972    )
 973    takeaway("BPE = repeatedly merge the most frequent adjacent pair; encoding replays the merges in order.")
 974
 975    banner("2. Byte-level BPE: no unknown tokens, ever")
 976    tok = trained_tokenizer()
 977    say(f"Trained a byte-level BPE on a small English corpus: {tok.vocab_size} tokens (256 bytes + {len(tok.merges)} merges).")
 978    samples = ["The password expires.", "def f(x): return x**2", "naïve café 🙂", "東京"]
 979    table(
 980        ["text", "tokens", "pieces", "round-trips?"],
 981        [(repr(s), len(tok.encode(s)), tok.tokens(s)[:8], tok.decode(tok.encode(s)) == s) for s in samples],
 982    )
 983    say(
 984        """
 985        Frequent English words are single tokens. Code, accents, emoji and
 986        Chinese never seen in training still encode: they fall back toward raw
 987        bytes. That costs more tokens but never fails.
 988        """
 989    )
 990
 991    banner("3. Quirks that explain odd model behavior")
 992    lead = tok.encode(" cat"), tok.encode("cat")
 993    say(f"Leading space: ' cat' -> ids {lead[0]}, 'cat' -> ids {lead[1]}. Different tokens, unrelated IDs.")
 994    say(
 995        f"""
 996        Numbers: ' 1234' -> {tok.tokens(' 1234')} but ' 12345' -> {tok.tokens(' 12345')},
 997        and '1234' at the start of a line -> {tok.tokens('1234')}. The same
 998        digits split differently depending on length and surroundings, one
 999        reason arithmetic is hard for LLMs.
1000        """
1001    )
1002    say(
1003        f"""
1004        Letters: ' strawberry' (with the leading space it has mid-sentence) ->
1005        {tok.tokens(' strawberry')}. The model sees
1006        {len(tok.encode(' strawberry'))} ids, not 10 letters, so 'how many r's?'
1007        is harder than it looks.
1008        """
1009    )
1010    languages = {
1011        "English": "The weather is nice today and we will go for a walk.",
1012        "German": "Das Wetter ist heute schön und wir gehen spazieren.",
1013        "Hindi": "आज मौसम अच्छा है और हम टहलने जाएंगे।",
1014    }
1015    table(
1016        ["language", "chars", "tokens", "chars/token"],
1017        [(k, len(v), len(tok.encode(v)), chars_per_token(tok, v)) for k, v in languages.items()],
1018        floatfmt=".2f",
1019    )
1020    say(
1021        """
1022        Same meaning, very different token counts. A tokenizer trained mostly on
1023        English has few merges for other scripts; Hindi stays near byte level
1024        (3 bytes per character). More tokens means more cost, more latency, and
1025        less content per context window.
1026        """
1027    )
1028
1029    banner("4. Rules of thumb for estimates")
1030    prose = "Tokens, not words, are the unit of cost, latency and context. " * 10
1031    say(
1032        f"""
1033        {len(prose)} characters of English ≈ {estimate_tokens(prose)} tokens at 4 chars/token.
1034        1,000 tokens ≈ {tokens_to_words(1000):.0f} words. A 2,000-token prompt with a
1035        500-token answer at example prices of $3 / $15 per million tokens costs
1036        ${estimate_cost(2000, 500, 3, 15):.4f}; a million such calls cost
1037        ${estimate_cost(2000, 500, 3, 15) * 1e6:,.0f}.
1038        """
1039    )
1040    takeaway(
1041        "Cost, latency and context limits are counted in tokens. English averages about 4 characters per token; "
1042        "code, JSON and non-English text need more. Measure with the real tokenizer."
1043    )
1044
1045
1046if __name__ == "__main__":
1047    demo()