What is a token? BPE and why some languages cost more
TL;DR
LLMs read tokens cut by algorithms like BPE, not letters. That explains why non-English text costs more and why models are poor at character-level tasks; measure tokens on real data and let code handle character work.
Why does the same content cost more tokens in Vietnamese than in English? Why does a model miscount the letter "r" in a word, or stumble when reversing a string? The answer lies in the first step of every LLM: tokenization, cutting text into pieces (tokens) before the model ever sees it. The model doesn't read letters; it reads a sequence of token IDs. Understanding this step helps you estimate cost correctly and avoid giving the model tasks it is blind to from the start.
What is a token?
A token is the smallest unit a model processes. It can be a whole common word ("the"), part of a word ("ing"), a space plus a word, or even a single byte. Each model has a fixed vocabulary of tens of thousands to a few hundred thousand tokens; every text is represented as a combination of tokens from it.
How does BPE work?
The most common algorithm is Byte Pair Encoding (BPE), brought into machine translation by Sennrich et al. in 2015 to handle rare words. The idea is simple:
- Start with a vocabulary of characters (or 256 bytes).
- Count the most frequent pair of adjacent symbols in the training data.
- Merge that pair into a new token and add it to the vocabulary.
- Repeat until the vocabulary reaches the desired size.
# Illustration: "low lower lowest"
l o w → merge (l,o) → lo w → merge (lo,w) → low
low e r → merge (e,r) → low erThe result: frequent strings become one token, rare strings are split into many small pieces. The karpathy/minbpe repo implements byte-level BPE in a few hundred very readable lines of Python; openai/tiktoken is the fast BPE tokenizer used for OpenAI's models.
Practical consequences
| Symptom | Cause in tokenization |
|---|---|
| Non-English text costs more tokens | Tokenizer training data skews toward English; accented characters are often split into several bytes/pieces |
| Counting characters and reversing strings go wrong | The model sees "strawberry" as 2–3 tokens, not individual letters |
| Arithmetic on long numbers is unreliable | Numbers are split into uneven groups of digits |
| An extra space changes the result | " hello" and "hello" are two different tokens |
| Same content, different price per model | Each model family has its own tokenizer and token counts |
Counting tokens properly
Don't rely on "1 token ≈ 4 characters" for non-English text; the error can be large. With Claude, use the API's token counting endpoint before sending:
count = client.messages.count_tokens(
model="claude-sonnet-5-5",
messages=[{"role": "user", "content": "Xin chào, bạn khỏe không?"}],
)
print(count.input_tokens)Counts from OpenAI's tokenizer (tiktoken) don't apply to Claude and vice versa, because each has its own vocabulary.
Risks and caveats
- Wrong cost estimates: quoting a project on English word counts and then running it on Vietnamese data can be significantly off. Measure on a sample of real data.
- Giving the model character-level work: counting letters, checking exact lengths, slicing strings by position… do these in code and let the model call a tool.
- "Weird" tokens in input: random strings (hashes, base64) are token-expensive and easy for the model to copy wrongly. Don't stuff them into prompts unless needed.
How to apply it
- Dev: Call the token counting API on real data samples before choosing a model and context size; handle character-level tasks in code.
- Tech Lead: Put "average tokens per document" by language into design docs; use it to compute cost and chunk limits for RAG.
- PM: When comparing prices across providers, compare on the same set of real texts, not mechanically on price per 1M tokens.
Sources & references
- Neural Machine Translation of Rare Words with Subword Units — Sennrich, Haddow, Birch (arXiv)
- minbpe — Andrej Karpathy (GitHub)
- tiktoken — OpenAI (GitHub)
- Token counting — Claude Docs