Data Science Academy - DSA

Data Science Academy - DSA At Data Science Academia, we are committed to provide a supportive and collaborative environment

16/10/2026

Type the same sentence in English and in Tamil. One costs 8 tokens. The other costs 53.

Same meaning. 6.6× the bill — and 6.6× the context window, gone.

Part 1 showed that AI reads text in chunks, not letters. This is where those chunks come from, and nobody chose them.

The tokenizer LEARNED them, by counting. The whole algorithm:
→ start with single characters
→ count every pair of neighbours
→ glue the commonest pair together
→ repeat 100,277 times

The first things it ever learned were "in", "er", "on", "re", "at". Not a design decision. Just what English looks like when you count it.

Which explains the weird stuff:
1234 splits as 123 + 4
12345 splits as 123 + 45
2024 splits as 202 + 4
The cut moves when the length does. It never sees a number.

And the part that costs you money: the tokenizer learned the scripts it saw most of. One Tamil cluster — க்கு — is FIVE tokens. One Chinese character is one. One Latin letter is one.

It isn't about being non-English — it's about what the tokenizer saw most of in training. One Chinese character costs 1 token. One Tamil cluster costs 5.

If your script wasn't well represented, you pay for it on every single call.

Address

No. 21, New State Bank Colony, West Tambaram
Chennai
600048

Opening Hours

Monday 9am - 6pm
Tuesday 9am - 6pm
Wednesday 9am - 6pm
Thursday 9am - 6pm
Friday 9am - 6pm
Saturday 9am - 6pm

Alerts

Be the first to know and let us send you an email when Data Science Academy - DSA posts news and promotions. Your email address will not be used for any other purpose, and you can unsubscribe at any time.

Shortcuts

Share

Nearby computer & electronics services