How Tokenization Powers (and Limits) Modern LLMs

When you type a prompt into ChatGPT, Claude, or Gemini, you’re not sending words. Your text is instantly broken into tokens — the fundamental units the model actually processes.

Why This Matters: Tokens Become Vectors

The model never sees language the way we do. It sees numbers. Every token gets converted into a vector (a list of hundreds of values) that places it in mathematical space.

• “King” and “Queen” sit close together.
• “King” and “pizza” are far apart.

The model understands meaning through distances and relationships in this vector space.

The Strawberry Problem

Ask any LLM: “How many R’s are in the word strawberry?” Most get it wrong.

Not because the model is stupid — because it never sees the full word. It sees “straw” and “berry” as two separate tokens and can’t properly count letters inside chunks it can’t see into.

How Token Vocabularies Are Built (BPE)

Almost every major model uses Byte Pair Encoding (BPE):

1. Start with every single character as its own token.
2. Repeatedly scan massive text corpora and merge the two most frequently adjacent tokens.
3. Repeat thousands of times until you get a vocabulary of ~32k–128k subword units.

This creates efficient chunks: “ing” becomes one token, “2026” gets its own slot, while rare words like “xylophone” get split.

The Hidden Language Bias

Most training data is heavily English (e.g. Llama 2 was ~89.7% English). Result:

• The same sentence in German/Italian costs ~50% more tokens than English.
• Languages like Burmese or Tibetan can cost up to 15× more.

This makes non-English prompts more expensive, slower, and less effective — a major fairness issue in AI.

2026 Reality: Multimodal Tokenization

Tokens aren’t just text anymore. Major models now handle a million tokens in a single conversation. Images, audio, and video all cost tokens too.

• An image becomes a grid of visual tokens (patches/tiles).
• Audio becomes tokens per second.
• Everything is fed into the same unified stream of numbers.

The model doesn’t “see” or “hear” — it just processes one long sequence of numerical tokens.

Practical Tips for Using AI Better

1. Front-load critical instructions (models process left-to-right with decaying attention).
2. Be concise — every word costs tokens and eats context.
3. Use clear naming in code (it tokenizes more efficiently).
4. For hard reasoning on smaller models, English still often outperforms most other languages.
5. Avoid character-level tasks (counting letters, spelling backwards, anagrams). Use AI for meaning, use code for precision.

The Future

Researchers are working on byte-level models (no tokenizer bias, but longer sequences) and adaptive multilingual tokenizers. The goal is equal performance and cost across all languages.