16/10/2026
Type the same sentence in English and in Tamil. One costs 8 tokens. The other costs 53.
Same meaning. 6.6× the bill — and 6.6× the context window, gone.
Part 1 showed that AI reads text in chunks, not letters. This is where those chunks come from, and nobody chose them.
The tokenizer LEARNED them, by counting. The whole algorithm:
→ start with single characters
→ count every pair of neighbours
→ glue the commonest pair together
→ repeat 100,277 times
The first things it ever learned were "in", "er", "on", "re", "at". Not a design decision. Just what English looks like when you count it.
Which explains the weird stuff:
1234 splits as 123 + 4
12345 splits as 123 + 45
2024 splits as 202 + 4
The cut moves when the length does. It never sees a number.
And the part that costs you money: the tokenizer learned the scripts it saw most of. One Tamil cluster — க்கு — is FIVE tokens. One Chinese character is one. One Latin letter is one.
It isn't about being non-English — it's about what the tokenizer saw most of in training. One Chinese character costs 1 token. One Tamil cluster costs 5.
If your script wasn't well represented, you pay for it on every single call.