A tokenizer decides how many tokens your text becomes, and so how much of the context window and the bill it uses.
How it works
A tokenizer has two parts: a fixed vocabulary that maps text pieces to integer IDs, and rules for splitting text into those pieces. Hugging Face’s documentation describes the common algorithms as subword methods: frequent words stay whole and rare words split into smaller known pieces, so “annoyingly” can become “annoying” + “ly”. Most current LLMs use byte pair encoding or a variant.
Encoding runs in three steps:
- Pre-tokenize. Split on spaces, punctuation and digits.
- Map pieces to IDs. Apply the learned merges and look up each piece in the vocabulary.
- Add special tokens. The chat template wraps messages in role markers, which also count.
Decoding reverses step 2. For open-weight models, all of this lives in a tokenizer.json file in the model repo.
Why it matters
Different tokenizers give different counts for the same text. We encoded the dataset’s texts with each model’s pinned tokenizer (English: Wikipedia and Project Gutenberg prose; Arabic: 2,009 FLORES-200 sentences plus three Wikipedia leads; measured 2026-11-15; tokens per 100 words = tokens / words x 100):
| Text | DeepSeek V4.1 Flash | GLM 5.3 | Qwen3.8 27B | Gemma 4 | GPT o200k |
|---|---|---|---|---|---|
| English prose, 1,183 words | 1,507 → 127.4 | 1,526 → 129.0 | 1,561 → 132.0 | 1,537 → 129.9 | 1,501 → 126.9 |
| Arabic MSA, 39,037 words | 86,690 → 222.1 | 95,230 → 243.9 | 71,424 → 183.0 | 79,120 → 202.7 | 73,593 → 188.5 |
English is close everywhere. Arabic is not: GLM 5.3 needs 95,230 / 71,424 = 1.33 times the tokens Qwen3.8 27B needs for the same text. On a per-token price, that gap is a cost gap, and it also changes how much Arabic fits in the context window. The Gemma column uses Google’s base Gemma 4 tokenizer; BroadNet’s build is based on that model, and we assume it keeps the tokenizer. The full per-model numbers, with CSV and JSON downloads, are in the tokenizer efficiency dataset.
On BroadNet Inference
BroadNet bills pay-as-you-go chat by input and output tokens and reports the counts in each response’s usage object, so the authoritative number is the one the API returns. To estimate before you send, paste text into the token counter or the Arabic token counter, which load the same pinned tokenizer files. The full measurement is in tokens per word in Arabic and English.
Common mistakes
- Counting with one model’s tokenizer and billing on another. Use the tokenizer of the model you call.
- Forgetting the chat template. Role markers and system prompts add tokens that plain-text counts miss.