Tokenizer efficiency across languages
Updated
1,000 words of English prose is about 1,290 tokens. The same 2,009 FLORES-200 sentences in Modern Standard Arabic need 1.25× to 3.03× as many tokens as in English, depending on the tokenizer. We measured 16 official tokenizers pinned to exact commits; counts, 95% intervals and method are free to download.
Download CSV Download JSON (with texts) Croissant metadata Counts CC BY 4.0, texts under their source licences, no FLORES-200 text · version 4.0 · measured 2026-11-15
Key findings
- 1,000 words of English prose is about 1,290 tokens, averaged over the five core tokenizers, or about 4.6 characters per token. Across all 16 tokenizers it ranges from 1,245 (MiniMax M3) to 1,320 (Qwen3.8 27B).
- Modern Standard Arabic needs 1.25× (Ministral 3 14B, 95% interval 1.24–1.25) to 3.03× (Phi-4, 3.01–3.05) the tokens of the same 2,009 FLORES-200 sentences in English. The dearest tokenizer needs 2.37× the tokens of the cheapest for the same Arabic.
- Arabic dialects in Arabic script: Egyptian Arabic 1.29× (Mistral Small 3.1) to 2.95× (Phi-4); Levantine Arabic 1.24× (Mistral Small 3.1) to 2.82× (Phi-4); Mesopotamian Arabic 1.26× (Ministral 3 14B) to 2.98× (Phi-4); Najdi Arabic 1.25× (Ministral 3 14B) to 3.03× (Phi-4); Gulf Arabic 1.15× (Ministral 3 14B) to 2.54× (Phi-4) the English token count. In FLORES-200, 1,339 of the 2,009 Najdi sentences (67%) are identical to the Modern Standard Arabic ones, so the Najdi figures are close to MSA by construction. Gulf Arabic has no FLORES-200 set; its 49 Tatoeba sentences give wide intervals (for example 1.13–1.29 on Qwen3.8).
- Arabizi, Arabic dialect typed in Latin letters and digits, costs 1.30× (Gemma 4 VL 26B) to 1.55× (Kimi K3) the English token count. The gap between tokenizers is narrower than for dialect in Arabic script (1.11× between dearest and cheapest, against 2.24×, 2.23×, 2.31×, 2.37×, 2.14×).
- Hindi needs 1.31× (Gemma 4 VL 26B, 95% interval 1.31–1.32) to 4.78× (Phi-4, 4.74–4.82) the English token count, and Urdu 1.53× (Gemma 4 VL 26B) to 4.36× (Phi-4). Hindi shows the widest gap between tokenizers: its dearest tokenizer needs 3.63× the tokens of its cheapest.
- Chinese needs 0.88× (Kimi K3) to 1.88× (Phi-4) the tokens of the same English sentences: tokenizers trained heavily on Chinese need fewer tokens for Chinese than for English.
- Code needs more tokens per character than prose: Python 3.7, JavaScript 3.7, SQL 3.8 characters per token on the five core tokenizers, against 4.6 for English prose. JSON ranges from 3,845 to 4,944 tokens per 1,000 words, and pretty-printed JSON uses 1.53× the tokens of the same JSON minified.
- Gemma 4 VL 26B and Gemma 3 27B, GPT (o200k) and gpt-oss (o200k_harmony), Ministral 3 14B and Nemotron 3 Ultra gave exactly the same count on all 52 texts, so in practice they share one vocabulary.
- On BroadNet's prices (2026-11-14), 1M characters of Arabic (MSA) input costs $0.0343 on BroadNet Gemma 4 VL 26B and $0.322 on GLM 5.3; English costs $0.0219 and $0.169. Every provider's price, by language, is in the real cost by language dataset.
How many more tokens does each language need than English?
Tokens for a set of sentences ÷ tokens for their English translations (same meaning). 1.00× means the language costs the same as English. 10 sets use the same 2,009 FLORES-200 sentences; Gulf Arabic and Arabizi use Tatoeba sentence pairs, because FLORES-200 has no Gulf Arabic or Arabizi. Under each ratio: its 95% bootstrap interval. One row per tokenizer; on a phone each tokenizer is a card.
| Tokenizer | Arabic MSA | Gulf Arabic (Tatoeba) | Egyptian Arabic | Levantine Arabic | Najdi Arabic | Mesopotamian Arabic | Arabizi (Tatoeba) | French | Spanish | Hindi | Urdu | Chinese |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek V4.1 Flash | 1.62× (1.61–1.63) | 1.44× (1.34–1.53) | 1.63× (1.61–1.64) | 1.58× (1.57–1.59) | 1.63× (1.61–1.64) | 1.62× (1.61–1.63) | 1.50× (1.40–1.61) | 1.54× (1.53–1.55) | 1.50× (1.48–1.51) | 2.81× (2.79–2.83) | 2.37× (2.36–2.39) | 0.95× (0.94–0.96) |
| GLM 5.3 | 1.77× (1.76–1.78) | 1.57× (1.47–1.68) | 1.75× (1.74–1.77) | 1.70× (1.68–1.71) | 1.77× (1.76–1.79) | 1.75× (1.74–1.77) | 1.48× (1.38–1.59) | 1.50× (1.49–1.51) | 1.44× (1.43–1.45) | 4.33× (4.29–4.37) | 2.75× (2.73–2.78) | 0.96× (0.95–0.97) |
| Qwen3.8 27B | 1.31× (1.30–1.32) | 1.21× (1.13–1.29) | 1.35× (1.34–1.36) | 1.29× (1.28–1.30) | 1.31× (1.30–1.32) | 1.32× (1.31–1.33) | 1.38× (1.29–1.47) | 1.38× (1.37–1.39) | 1.32× (1.31–1.33) | 2.02× (2.00–2.03) | 1.98× (1.97–2.00) | 0.92× (0.91–0.93) |
| Gemma 4 VL 26B | 1.47× (1.46–1.48) | 1.22× (1.12–1.32) | 1.44× (1.42–1.45) | 1.41× (1.40–1.43) | 1.47× (1.46–1.49) | 1.46× (1.45–1.47) | 1.30× (1.21–1.39) | 1.40× (1.39–1.41) | 1.29× (1.29–1.30) | 1.31× (1.31–1.32) | 1.53× (1.52–1.54) | 1.09× (1.08–1.10) |
| GPT (o200k) | 1.38× (1.37–1.39) | 1.32× (1.22–1.42) | 1.41× (1.40–1.42) | 1.35× (1.34–1.36) | 1.39× (1.38–1.40) | 1.39× (1.38–1.40) | 1.45× (1.35–1.55) | 1.36× (1.35–1.37) | 1.32× (1.31–1.33) | 1.58× (1.56–1.59) | 1.66× (1.65–1.67) | 1.26× (1.25–1.27) |
| gpt-oss (o200k_harmony) | 1.38× (1.37–1.39) | 1.32× (1.22–1.42) | 1.41× (1.40–1.42) | 1.35× (1.34–1.36) | 1.39× (1.38–1.40) | 1.39× (1.38–1.40) | 1.45× (1.35–1.55) | 1.36× (1.35–1.37) | 1.32× (1.31–1.33) | 1.58× (1.56–1.59) | 1.66× (1.65–1.67) | 1.26× (1.25–1.27) |
| Llama 3.3 70B | 1.66× (1.64–1.67) | 1.50× (1.40–1.60) | 1.65× (1.64–1.66) | 1.59× (1.58–1.60) | 1.66× (1.65–1.67) | 1.65× (1.64–1.66) | 1.49× (1.39–1.59) | 1.59× (1.58–1.60) | 1.54× (1.53–1.55) | 2.53× (2.51–2.55) | 3.09× (3.06–3.11) | 1.29× (1.28–1.30) |
| Llama 4 Maverick | 1.59× (1.58–1.60) | 1.45× (1.35–1.55) | 1.61× (1.59–1.62) | 1.55× (1.54–1.57) | 1.60× (1.58–1.61) | 1.59× (1.58–1.60) | 1.46× (1.36–1.55) | 1.36× (1.35–1.37) | 1.30× (1.29–1.31) | 1.65× (1.64–1.66) | 1.89× (1.88–1.90) | 1.07× (1.06–1.08) |
| Kimi K3 | 2.05× (2.04–2.07) | 1.86× (1.75–1.97) | 2.02× (2.01–2.04) | 1.94× (1.92–1.95) | 2.06× (2.04–2.07) | 2.03× (2.02–2.05) | 1.55× (1.44–1.66) | 1.72× (1.71–1.74) | 1.69× (1.68–1.71) | 2.75× (2.73–2.77) | 2.17× (2.16–2.19) | 0.88× (0.87–0.89) |
| MiniMax M3 | 1.36× (1.35–1.37) | 1.32× (1.23–1.43) | 1.40× (1.39–1.41) | 1.33× (1.32–1.34) | 1.36× (1.35–1.37) | 1.37× (1.36–1.38) | 1.51× (1.40–1.62) | 1.29× (1.28–1.30) | 1.25× (1.24–1.26) | 2.94× (2.92–2.96) | 3.78× (3.74–3.81) | 0.92× (0.91–0.93) |
| Gemma 3 27B | 1.47× (1.46–1.48) | 1.22× (1.12–1.32) | 1.44× (1.42–1.45) | 1.41× (1.40–1.43) | 1.47× (1.46–1.49) | 1.46× (1.45–1.47) | 1.30× (1.21–1.39) | 1.40× (1.39–1.41) | 1.29× (1.29–1.30) | 1.31× (1.31–1.32) | 1.53× (1.52–1.54) | 1.09× (1.08–1.10) |
| Phi-4 | 3.03× (3.01–3.05) | 2.54× (2.37–2.71) | 2.95× (2.93–2.98) | 2.82× (2.80–2.84) | 3.03× (3.01–3.06) | 2.98× (2.96–3.00) | 1.51× (1.40–1.61) | 1.60× (1.58–1.61) | 1.55× (1.53–1.56) | 4.78× (4.74–4.82) | 4.36× (4.33–4.40) | 1.88× (1.86–1.90) |
| Mistral Small 3.1 | 1.27× (1.26–1.28) | 1.17× (1.09–1.25) | 1.29× (1.28–1.30) | 1.24× (1.23–1.25) | 1.27× (1.26–1.28) | 1.27× (1.26–1.28) | 1.44× (1.33–1.55) | 1.31× (1.30–1.32) | 1.32× (1.31–1.33) | 2.76× (2.74–2.79) | 1.76× (1.75–1.78) | 1.42× (1.40–1.43) |
| Ministral 3 14B | 1.25× (1.24–1.25) | 1.15× (1.08–1.23) | 1.29× (1.28–1.30) | 1.24× (1.23–1.24) | 1.25× (1.24–1.26) | 1.26× (1.25–1.27) | 1.44× (1.33–1.55) | 1.31× (1.30–1.32) | 1.32× (1.31–1.33) | 1.80× (1.78–1.81) | 1.76× (1.75–1.77) | 1.42× (1.40–1.43) |
| Nemotron 3 Ultra | 1.25× (1.24–1.25) | 1.15× (1.08–1.23) | 1.29× (1.28–1.30) | 1.24× (1.23–1.24) | 1.25× (1.24–1.26) | 1.26× (1.25–1.27) | 1.44× (1.33–1.55) | 1.31× (1.30–1.32) | 1.32× (1.31–1.33) | 1.80× (1.78–1.81) | 1.76× (1.75–1.77) | 1.42× (1.40–1.43) |
| Qwen3 32B | 1.63× (1.62–1.64) | 1.47× (1.36–1.59) | 1.64× (1.62–1.65) | 1.58× (1.57–1.59) | 1.63× (1.62–1.65) | 1.62× (1.61–1.64) | 1.50× (1.39–1.60) | 1.57× (1.56–1.58) | 1.52× (1.51–1.53) | 4.43× (4.39–4.47) | 3.16× (3.14–3.19) | 1.01× (1.00–1.02) |
Sentences per set: Arabic MSA 2,009 (FLORES-200), Gulf Arabic 49 (Tatoeba), Egyptian Arabic 2,009 (FLORES-200), Levantine Arabic 2,009 (FLORES-200), Najdi Arabic 2,009 (FLORES-200), Mesopotamian Arabic 2,009 (FLORES-200), Arabizi 39 (Tatoeba), French 2,009 (FLORES-200), Spanish 2,009 (FLORES-200), Hindi 2,009 (FLORES-200), Urdu 2,009 (FLORES-200), Chinese 2,009 (FLORES-200). Interval: percentile bootstrap over sentence pairs, seeded mulberry32 PRNG, 10,000 resamples, seed 20261115.
How many tokens is 1,000 words in each language?
All texts of a kind pooled: total tokens ÷ total words × 1,000. English is the English prose texts (Wikipedia and Project Gutenberg), not FLORES-200; the other languages pool their FLORES-200 sentences with their Wikipedia leads, so FLORES-200 text dominates them. Words are word-like segments from Intl.Segmenter; for Chinese these are segmented words, not characters. Characters per token for every tokenizer and text kind are in the CSV and JSON.
| Tokenizer | English | Arabic MSA | Egyptian Arabic | Levantine Arabic | Najdi Arabic | Mesopotamian Arabic | Gulf Arabic | Arabizi | French | Spanish | Hindi | Urdu | Chinese |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek V4.1 Flash | 1,274 | 2,221 | 2,251 | 2,370 | 2,225 | 2,321 | 2,315 | 2,276 | 1,635 | 1,548 | 2,934 | 2,302 | 1,124 |
| GLM 5.3 | 1,290 | 2,439 | 2,437 | 2,562 | 2,441 | 2,531 | 2,507 | 2,248 | 1,604 | 1,500 | 4,547 | 2,683 | 1,138 |
| Qwen3.8 27B | 1,320 | 1,830 | 1,897 | 1,972 | 1,832 | 1,930 | 2,212 | 2,336 | 1,499 | 1,394 | 2,149 | 1,960 | 1,109 |
| Gemma 4 VL 26B | 1,299 | 2,027 | 1,997 | 2,137 | 2,029 | 2,107 | 2,256 | 2,276 | 1,494 | 1,347 | 1,380 | 1,488 | 1,292 |
| GPT (o200k) | 1,269 | 1,885 | 1,937 | 2,019 | 1,887 | 1,977 | 2,049 | 2,098 | 1,436 | 1,355 | 1,637 | 1,599 | 1,478 |
| gpt-oss (o200k_harmony) | 1,269 | 1,885 | 1,937 | 2,019 | 1,887 | 1,977 | 2,049 | 2,098 | 1,436 | 1,355 | 1,637 | 1,599 | 1,478 |
| Llama 3.3 70B | 1,289 | 2,283 | 2,291 | 2,405 | 2,287 | 2,376 | 2,394 | 2,248 | 1,698 | 1,604 | 2,653 | 3,006 | 1,534 |
| Llama 4 Maverick | 1,264 | 2,186 | 2,223 | 2,337 | 2,186 | 2,283 | 2,281 | 2,131 | 1,444 | 1,343 | 1,726 | 1,833 | 1,269 |
| Kimi K3 | 1,292 | 2,830 | 2,807 | 2,921 | 2,829 | 2,927 | 2,911 | 2,262 | 1,839 | 1,760 | 2,884 | 2,114 | 1,046 |
| MiniMax M3 | 1,245 | 1,828 | 1,908 | 1,969 | 1,831 | 1,930 | 2,054 | 2,196 | 1,350 | 1,274 | 3,016 | 3,599 | 1,069 |
| Gemma 3 27B | 1,299 | 2,027 | 1,997 | 2,137 | 2,029 | 2,107 | 2,256 | 2,276 | 1,494 | 1,347 | 1,380 | 1,488 | 1,292 |
| Phi-4 | 1,289 | 4,177 | 4,103 | 4,262 | 4,174 | 4,297 | 4,049 | 2,280 | 1,705 | 1,607 | 5,013 | 4,253 | 2,232 |
| Mistral Small 3.1 | 1,302 | 1,790 | 1,838 | 1,923 | 1,792 | 1,879 | 1,916 | 2,192 | 1,437 | 1,404 | 2,974 | 1,762 | 1,727 |
| Ministral 3 14B | 1,302 | 1,760 | 1,836 | 1,913 | 1,763 | 1,864 | 1,892 | 2,192 | 1,437 | 1,404 | 1,934 | 1,758 | 1,727 |
| Nemotron 3 Ultra | 1,302 | 1,760 | 1,836 | 1,913 | 1,763 | 1,864 | 1,892 | 2,192 | 1,437 | 1,404 | 1,934 | 1,758 | 1,727 |
| Qwen3 32B | 1,306 | 2,285 | 2,312 | 2,427 | 2,285 | 2,380 | 2,350 | 2,271 | 1,705 | 1,611 | 4,729 | 3,136 | 1,227 |
Code and JSON
| Tokenizer | Python tokens / 1k words | JavaScript tokens / 1k words | SQL tokens / 1k words | JSON tokens / 1k words | Python chars / token | JavaScript chars / token | SQL chars / token | JSON chars / token |
|---|---|---|---|---|---|---|---|---|
| DeepSeek V4.1 Flash | 2,107 | 1,796 | 2,232 | 4,010 | 3.76 | 3.76 | 3.85 | 3.09 |
| GLM 5.3 | 2,004 | 1,788 | 2,098 | 4,114 | 3.95 | 3.79 | 4.10 | 3.00 |
| Qwen3.8 27B | 2,154 | 1,937 | 2,347 | 4,839 | 3.68 | 3.48 | 3.66 | 2.56 |
| Gemma 4 VL 26B | 2,339 | 1,986 | 2,448 | 4,944 | 3.38 | 3.40 | 3.51 | 2.50 |
| GPT (o200k) | 2,025 | 1,755 | 2,115 | 3,888 | 3.91 | 3.85 | 4.07 | 3.17 |
| gpt-oss (o200k_harmony) | 2,025 | 1,755 | 2,115 | 3,888 | 3.91 | 3.85 | 4.07 | 3.17 |
| Llama 3.3 70B | 2,004 | 1,788 | 2,098 | 3,888 | 3.95 | 3.79 | 4.10 | 3.17 |
| Llama 4 Maverick | 2,025 | 1,755 | 2,151 | 3,845 | 3.91 | 3.85 | 3.98 | 3.22 |
| Kimi K3 | 2,050 | 1,790 | 2,176 | 3,889 | 3.86 | 3.77 | 3.94 | 3.17 |
| MiniMax M3 | 2,061 | 1,757 | 2,176 | 3,899 | 3.83 | 3.85 | 3.94 | 3.17 |
| Gemma 3 27B | 2,339 | 1,986 | 2,448 | 4,944 | 3.38 | 3.40 | 3.51 | 2.50 |
| Phi-4 | 2,004 | 1,788 | 2,098 | 4,017 | 3.95 | 3.79 | 4.10 | 3.08 |
| Mistral Small 3.1 | 2,096 | 1,820 | 2,381 | 4,743 | 3.77 | 3.72 | 3.61 | 2.60 |
| Ministral 3 14B | 2,096 | 1,788 | 2,381 | 4,725 | 3.77 | 3.79 | 3.61 | 2.62 |
| Nemotron 3 Ultra | 2,096 | 1,788 | 2,381 | 4,725 | 3.77 | 3.79 | 3.61 | 2.62 |
| Qwen3 32B | 2,018 | 1,812 | 2,185 | 4,631 | 3.92 | 3.73 | 3.92 | 2.67 |
Which tokenizers are measured?
Each is the tokenizer file the model maker publishes, pinned to a commit. The first five (DeepSeek, GLM, Qwen3.8, Gemma 4 and GPT o200k) are the v2 core set behind the site's token tools; the other 11 were added in version 3. Which priced models use each tokenizer is checked file by file in the real cost by language dataset.
| Tokenizer | Maker | Vocabulary | File and commit | Note |
|---|---|---|---|---|
| DeepSeek V4.1 Flash · core | DeepSeek | 128,000 | deepseek-ai/DeepSeek-V4.1-Flash @ 314113f | Official tokenizer.json from the DeepSeek-V4.1-Flash model repo. |
| GLM 5.3 · core | Z.ai | 154,820 | zai-org/GLM-5.3 @ ba32f8c | Official tokenizer.json from the zai-org/GLM-5.3 model repo. |
| Qwen3.8 27B · core | Qwen | 248,044 | Qwen/Qwen3.8-27B @ 3756181 | Official tokenizer.json from the Qwen/Qwen3.8-27B model repo. |
| Gemma 4 VL 26B · core | 262,144 | google/gemma-4-26B-A4B-it @ 4d7ae49 | Google's Gemma 4 26B A4B tokenizer. BroadNet's build is based on this model; we assume it keeps the base tokenizer. | |
| GPT (o200k) · core | OpenAI | 199,998 | o200k_base (gpt-tokenizer) | o200k_base, the encoding used by GPT-4o, GPT-4.1, o-series and GPT-5 models (via gpt-tokenizer). Reference only: not sold by BroadNet. |
| gpt-oss (o200k_harmony) | OpenAI | 199,998 | openai/gpt-oss-120b @ b5c939d | o200k_harmony is o200k_base plus the Harmony chat-format special tokens, so ordinary text should count the same as GPT (o200k). |
| Llama 3.3 70B | Meta | 128,000 | meta-llama/Llama-3.3-70B-Instruct @ 6f6073b | Meta's repo is gated. Bytes from the unsloth mirror, proven identical to Meta's tokenizer.json at the pinned commit. Proof: SHA-256 6b9e4e7fb171… rebuilds the LFS pointer with Git object 1c1d8d5c9024, the ID meta-llama/Llama-3.3-70B-Instruct lists for tokenizer.json. |
| Llama 4 Maverick | Meta | 200,000 | meta-llama/Llama-4-Maverick-17B-128E-Instruct @ 73d1471 | Meta's repo is gated. Bytes from the unsloth mirror, proven identical to Meta's tokenizer.json at the pinned commit. Proof: SHA-256 172c9eb4beaf… rebuilds the LFS pointer with Git object b1fde397c877, the ID meta-llama/Llama-4-Maverick-17B-128E-Instruct lists for tokenizer.json. |
| Kimi K3 | Moonshot AI | 163,584 | moonshotai/Kimi-K3 @ f831ab6 | Moonshot publishes no tokenizer.json: the official tokenizer is the tiktoken rank file plus the regex in tokenization_kimi.py. Encoded with a tiktoken-compatible counter that matched Python tiktoken on every corpus text. |
| MiniMax M3 | MiniMax | 200,000 | MiniMaxAI/MiniMax-M3 @ d95e30c | Official tokenizer.json from the MiniMaxAI/MiniMax-M3 model repo. |
| Gemma 3 27B | 262,144 | google/gemma-3-27b-it @ 005ad34 | Google's repo is gated. Bytes from the unsloth mirror, proven identical to Google's tokenizer.json at the pinned commit. Proof: SHA-256 4667f2089529… rebuilds the LFS pointer with Git object 29401f984828, the ID google/gemma-3-27b-it lists for tokenizer.json. | |
| Phi-4 | Microsoft | 100,352 | microsoft/phi-4 @ 2db69c1 | Official tokenizer.json from microsoft/phi-4. Phi-4-mini and Phi-4-reasoning ship different tokenizer files and are not mapped to it. |
| Mistral Small 3.1 | Mistral AI | 131,072 | mistralai/Mistral-Small-3.1-24B-Instruct-2503 @ 68faf51 | Official tokenizer.json from the Mistral Small 3.1 repo. |
| Ministral 3 14B | Mistral AI | 131,072 | mistralai/Ministral-3-14B-Instruct-2512 @ 29439f8 | Official tokenizer.json from the Ministral 3 14B repo; the 3B and 8B repos ship the same file. |
| Nemotron 3 Ultra | NVIDIA | 131,072 | nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 @ 77df655 | Official tokenizer.json from NVIDIA's Nemotron 3 Ultra repo. |
| Qwen3 32B | Qwen | 151,643 | Qwen/Qwen3-32B @ 9216db5 | The original Qwen3 tokenizer. Qwen3.5 and later use the larger Qwen3.8 vocabulary, measured separately. |
What does 1M characters cost on each BroadNet model?
Input cost of one million characters of each kind of text = tokens per character × the model's input price. Prices: BroadNet Inference rate card, 2026-11-14 (DeepSeek V4.1 Flash $0.175, GLM 5.3 $0.78, Qwen3.8 27B $0.175, BroadNet Gemma 4 VL 26B $0.10 per 1M input tokens). They update with the rate card. For every provider that sells a model with a measured tokenizer, see real cost by language.
| Text kind | DeepSeek V4.1 Flash | GLM 5.3 | Qwen3.8 27B | BroadNet Gemma 4 VL 26B |
|---|---|---|---|---|
| English prose | $0.0375 | $0.169 | $0.0388 | $0.0219 |
| English, FLORES-200 sentences | $0.0359 | $0.161 | $0.0365 | $0.0206 |
| Arabic (Modern Standard) | $0.0658 | $0.322 | $0.0542 | $0.0343 |
| Gulf Arabic | $0.0725 | $0.350 | $0.0693 | $0.0404 |
| Egyptian Arabic | $0.0677 | $0.327 | $0.0570 | $0.0343 |
| Levantine Arabic (North) | $0.0689 | $0.332 | $0.0573 | $0.0355 |
| Najdi Arabic (central Saudi Arabia) | $0.0660 | $0.323 | $0.0543 | $0.0344 |
| Mesopotamian Arabic (Iraq) | $0.0673 | $0.327 | $0.0559 | $0.0349 |
| Arabizi (Arabic in Latin letters) | $0.0736 | $0.324 | $0.0756 | $0.0421 |
| French | $0.0463 | $0.203 | $0.0425 | $0.0242 |
| Spanish | $0.0451 | $0.195 | $0.0406 | $0.0224 |
| Hindi | $0.101 | $0.698 | $0.0740 | $0.0272 |
| Urdu | $0.0860 | $0.447 | $0.0732 | $0.0318 |
| Chinese (Simplified) | $0.103 | $0.463 | $0.101 | $0.0674 |
| Code: Python | $0.0466 | $0.198 | $0.0477 | $0.0296 |
| Code: JavaScript | $0.0465 | $0.206 | $0.0501 | $0.0294 |
| Code: SQL | $0.0455 | $0.191 | $0.0479 | $0.0285 |
| JSON | $0.0568 | $0.260 | $0.0685 | $0.0400 |
A cheaper per-token price can still cost more per character if the tokenizer splits that language finely. Output prices per 1M characters are in the JSON download.
Every text in the dataset
Raw token counts for all 52 texts and 16 tokenizers (wide table)
| Text | Licence | Words | Chars | DeepSeek V4.1 Flash | GLM 5.3 | Qwen3.8 27B | Gemma 4 VL 26B | GPT (o200k) | gpt-oss (o200k_harmony) | Llama 3.3 70B | Llama 4 Maverick | Kimi K3 | MiniMax M3 | Gemma 3 27B | Phi-4 | Mistral Small 3.1 | Ministral 3 14B | Nemotron 3 Ultra | Qwen3 32B |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| English, FLORES-200 sentences | CC BY-SA 4.0 | 43,194 | 257,160 | 52,736 | 53,004 | 53,698 | 52,995 | 52,409 | 52,409 | 52,963 | 52,760 | 52,921 | 51,821 | 52,995 | 52,983 | 54,305 | 54,305 | 54,305 | 53,876 |
| Modern Standard Arabic, FLORES-200 sentences | CC BY-SA 4.0 | 38,453 | 226,913 | 85,363 | 93,708 | 70,314 | 77,910 | 72,437 | 72,437 | 87,721 | 83,999 | 108,728 | 70,242 | 77,910 | 160,527 | 68,776 | 67,646 | 67,646 | 87,789 |
| Egyptian Arabic, FLORES-200 sentences | CC BY-SA 4.0 | 38,106 | 221,934 | 85,799 | 92,879 | 72,237 | 76,080 | 73,786 | 73,786 | 87,322 | 84,743 | 107,041 | 72,691 | 76,080 | 156,492 | 70,003 | 69,919 | 69,919 | 88,118 |
| North Levantine Arabic, FLORES-200 sentences | CC BY-SA 4.0 | 35,084 | 211,226 | 83,136 | 89,888 | 69,198 | 74,980 | 70,827 | 70,827 | 84,371 | 82,001 | 102,487 | 69,065 | 74,980 | 149,543 | 67,459 | 67,112 | 67,112 | 85,152 |
| Najdi Arabic, FLORES-200 sentences | CC BY-SA 4.0 | 38,508 | 227,152 | 85,699 | 94,000 | 70,529 | 78,116 | 72,672 | 72,672 | 88,062 | 84,193 | 108,925 | 70,499 | 78,116 | 160,743 | 69,024 | 67,904 | 67,904 | 87,999 |
| Mesopotamian Arabic, FLORES-200 sentences | CC BY-SA 4.0 | 36,747 | 221,971 | 85,306 | 92,994 | 70,909 | 77,418 | 72,638 | 72,638 | 87,313 | 83,898 | 107,543 | 70,912 | 77,418 | 157,920 | 69,031 | 68,510 | 68,510 | 87,448 |
| Hindi, FLORES-200 sentences | CC BY-SA 4.0 | 50,520 | 256,462 | 148,122 | 229,555 | 108,371 | 69,629 | 82,569 | 82,569 | 133,935 | 87,034 | 145,630 | 152,280 | 69,629 | 253,044 | 150,108 | 97,510 | 97,510 | 238,687 |
| Urdu, FLORES-200 sentences | CC BY-SA 4.0 | 54,365 | 254,550 | 125,216 | 145,930 | 106,567 | 80,891 | 86,942 | 86,942 | 163,447 | 99,684 | 114,949 | 195,676 | 80,891 | 231,239 | 95,776 | 95,597 | 95,597 | 170,501 |
| Chinese (Simplified), FLORES-200 sentences | CC BY-SA 4.0 | 44,517 | 85,435 | 50,085 | 50,712 | 49,394 | 57,545 | 65,875 | 65,875 | 68,361 | 56,574 | 46,594 | 47,682 | 57,545 | 99,570 | 76,890 | 76,890 | 76,890 | 54,612 |
| French, FLORES-200 sentences | CC BY-SA 4.0 | 49,654 | 306,479 | 81,069 | 79,507 | 74,329 | 74,101 | 71,183 | 71,183 | 84,200 | 71,564 | 91,255 | 66,930 | 74,101 | 84,558 | 71,240 | 71,240 | 71,240 | 84,549 |
| Spanish, FLORES-200 sentences | CC BY-SA 4.0 | 50,973 | 306,274 | 78,871 | 76,416 | 70,988 | 68,625 | 68,999 | 68,999 | 81,747 | 68,414 | 89,690 | 64,850 | 68,625 | 81,905 | 71,495 | 71,495 | 71,495 | 82,085 |
| Gulf Arabic, Tatoeba sentences | CC BY 2.0 FR | 203 | 1,134 | 470 | 509 | 449 | 458 | 416 | 416 | 486 | 463 | 591 | 417 | 458 | 822 | 389 | 384 | 384 | 477 |
| English side of Gulf Arabic, Tatoeba sentences | CC BY 2.0 FR | 260 | 1,414 | 327 | 324 | 372 | 376 | 316 | 316 | 324 | 320 | 318 | 315 | 376 | 324 | 333 | 333 | 333 | 324 |
| Arabizi (Arabic dialects in Latin letters), Tatoeba sentences | CC BY 2.0 FR | 214 | 1,158 | 487 | 481 | 500 | 487 | 449 | 449 | 481 | 456 | 484 | 470 | 487 | 488 | 469 | 469 | 469 | 486 |
| English side of Arabizi (Arabic dialects in Latin letters), Tatoeba sentences | CC BY 2.0 FR | 260 | 1,294 | 324 | 324 | 363 | 375 | 310 | 310 | 323 | 313 | 312 | 311 | 375 | 324 | 325 | 325 | 325 | 325 |
| English Wikipedia: Water | CC BY-SA 4.0 | 274 | 1,706 | 351 | 364 | 370 | 356 | 355 | 355 | 364 | 348 | 362 | 346 | 356 | 364 | 361 | 361 | 361 | 368 |
| Arabic (Modern Standard) Wikipedia: ماء | CC BY-SA 4.0 | 222 | 1,346 | 507 | 588 | 435 | 473 | 443 | 443 | 534 | 508 | 667 | 433 | 473 | 944 | 425 | 419 | 419 | 547 |
| Egyptian Arabic Wikipedia: ميه | CC BY-SA 4.0 | 78 | 419 | 194 | 196 | 180 | 180 | 178 | 178 | 194 | 185 | 221 | 177 | 180 | 290 | 175 | 175 | 175 | 203 |
| French Wikipedia: Eau | CC BY-SA 4.0 | 249 | 1,620 | 461 | 458 | 434 | 409 | 429 | 429 | 483 | 418 | 504 | 385 | 409 | 484 | 412 | 412 | 412 | 484 |
| Spanish Wikipedia: Agua | CC BY-SA 4.0 | 205 | 1,229 | 335 | 338 | 321 | 300 | 305 | 305 | 364 | 296 | 390 | 285 | 300 | 365 | 316 | 316 | 316 | 370 |
| Hindi Wikipedia: जल | CC BY-SA 4.0 | 321 | 1,575 | 910 | 1,408 | 717 | 493 | 569 | 569 | 826 | 603 | 888 | 929 | 493 | 1,543 | 908 | 652 | 652 | 1,460 |
| Chinese Wikipedia: 水 | CC BY-SA 4.0 | 276 | 505 | 288 | 302 | 297 | 355 | 420 | 420 | 419 | 323 | 275 | 275 | 355 | 584 | 482 | 482 | 482 | 334 |
| Urdu Wikipedia: پانی | CC BY-SA 4.0 | 103 | 489 | 246 | 295 | 231 | 198 | 196 | 196 | 343 | 216 | 235 | 394 | 198 | 451 | 215 | 215 | 215 | 346 |
| English Wikipedia: Moon | CC BY-SA 4.0 | 365 | 2,149 | 464 | 472 | 484 | 479 | 466 | 466 | 471 | 462 | 469 | 458 | 479 | 471 | 484 | 484 | 484 | 485 |
| Arabic (Modern Standard) Wikipedia: القمر | CC BY-SA 4.0 | 251 | 1,460 | 552 | 621 | 455 | 493 | 472 | 472 | 581 | 553 | 715 | 460 | 493 | 1,052 | 453 | 437 | 437 | 584 |
| Egyptian Arabic Wikipedia: قمر | CC BY-SA 4.0 | 276 | 1,480 | 569 | 647 | 535 | 540 | 542 | 542 | 593 | 583 | 705 | 527 | 540 | 1,024 | 508 | 508 | 508 | 626 |
| French Wikipedia: Lune | CC BY-SA 4.0 | 342 | 2,087 | 545 | 557 | 506 | 498 | 498 | 498 | 577 | 488 | 611 | 459 | 498 | 577 | 491 | 491 | 491 | 585 |
| Spanish Wikipedia: Luna | CC BY-SA 4.0 | 243 | 1,372 | 348 | 355 | 320 | 307 | 309 | 309 | 368 | 305 | 394 | 289 | 307 | 368 | 322 | 322 | 322 | 368 |
| Hindi Wikipedia: चन्द्रमा | CC BY-SA 4.0 | 306 | 1,542 | 854 | 1,334 | 687 | 425 | 507 | 507 | 811 | 555 | 828 | 851 | 425 | 1,483 | 906 | 626 | 626 | 1,413 |
| Chinese Wikipedia: 月球 | CC BY-SA 4.0 | 605 | 1,087 | 668 | 675 | 682 | 803 | 877 | 877 | 911 | 755 | 624 | 625 | 803 | 1,244 | 1,027 | 1,027 | 1,027 | 772 |
| Urdu Wikipedia: چاند | CC BY-SA 4.0 | 264 | 1,162 | 533 | 669 | 469 | 350 | 379 | 379 | 781 | 436 | 510 | 931 | 350 | 1,112 | 415 | 412 | 412 | 798 |
| English Wikipedia: Internet | CC BY-SA 4.0 | 189 | 1,338 | 234 | 233 | 234 | 235 | 233 | 233 | 233 | 233 | 233 | 232 | 235 | 233 | 235 | 235 | 235 | 233 |
| Arabic (Modern Standard) Wikipedia: إنترنت | CC BY-SA 4.0 | 111 | 767 | 268 | 313 | 220 | 244 | 241 | 241 | 281 | 277 | 355 | 226 | 244 | 534 | 223 | 215 | 215 | 288 |
| Egyptian Arabic Wikipedia: انترنت | CC BY-SA 4.0 | 87 | 551 | 189 | 203 | 156 | 162 | 160 | 160 | 195 | 187 | 238 | 162 | 162 | 353 | 161 | 161 | 161 | 191 |
| French Wikipedia: Internet | CC BY-SA 4.0 | 182 | 1,228 | 315 | 304 | 293 | 292 | 279 | 279 | 327 | 278 | 350 | 265 | 292 | 327 | 294 | 294 | 294 | 336 |
| Spanish Wikipedia: Internet | CC BY-SA 4.0 | 194 | 1,254 | 323 | 300 | 283 | 272 | 271 | 271 | 318 | 263 | 350 | 260 | 272 | 318 | 300 | 300 | 300 | 317 |
| Hindi Wikipedia: अंतरजाल | CC BY-SA 4.0 | 339 | 2,012 | 1,186 | 1,826 | 878 | 518 | 622 | 622 | 1,033 | 658 | 1,156 | 1,220 | 518 | 2,040 | 1,173 | 803 | 803 | 1,899 |
| Chinese Wikipedia: 互联网 | CC BY-SA 4.0 | 251 | 471 | 250 | 253 | 257 | 293 | 317 | 317 | 333 | 272 | 236 | 236 | 293 | 493 | 438 | 438 | 438 | 280 |
| Urdu Wikipedia: انٹرنیٹ | CC BY-SA 4.0 | 143 | 775 | 314 | 362 | 275 | 210 | 237 | 237 | 389 | 268 | 325 | 472 | 210 | 606 | 259 | 256 | 256 | 444 |
| Python code (Wikipedia: Python syntax and semantics) | CC BY-SA 4.0 | 208 | 1,706 | 452 | 427 | 453 | 501 | 427 | 427 | 427 | 427 | 434 | 436 | 501 | 427 | 446 | 446 | 446 | 431 |
| Python code (Wikipedia: Python (programming language)) | CC BY-SA 4.0 | 72 | 508 | 138 | 134 | 150 | 154 | 140 | 140 | 134 | 140 | 140 | 141 | 154 | 134 | 141 | 141 | 141 | 134 |
| JavaScript code (Wikipedia: JavaScript syntax) | CC BY-SA 4.0 | 238 | 1,513 | 436 | 434 | 461 | 470 | 426 | 426 | 434 | 426 | 434 | 427 | 470 | 434 | 439 | 431 | 431 | 436 |
| JavaScript code (Wikipedia: JavaScript) | CC BY-SA 4.0 | 252 | 1,799 | 444 | 442 | 488 | 503 | 434 | 434 | 442 | 434 | 443 | 434 | 503 | 442 | 453 | 445 | 445 | 452 |
| SQL code (Wikipedia: Select (SQL)) | CC BY-SA 4.0 | 212 | 1,509 | 460 | 436 | 475 | 497 | 439 | 439 | 436 | 437 | 440 | 442 | 497 | 436 | 476 | 476 | 476 | 456 |
| SQL code (Wikipedia: Join (SQL)) | CC BY-SA 4.0 | 145 | 1,555 | 337 | 313 | 363 | 377 | 316 | 316 | 313 | 331 | 337 | 335 | 377 | 313 | 374 | 374 | 374 | 324 |
| English novel: Pride and Prejudice (Austen, 1813) | Public domain (USA) | 355 | 1,841 | 458 | 457 | 473 | 467 | 447 | 447 | 457 | 452 | 465 | 437 | 467 | 457 | 460 | 460 | 460 | 459 |
| French novel: Candide (Voltaire, 1759) | Public domain (USA) | 318 | 1,908 | 565 | 547 | 518 | 515 | 503 | 503 | 565 | 509 | 618 | 474 | 515 | 565 | 488 | 488 | 488 | 563 |
| Spanish novel: Don Quijote (Cervantes, 1605) | Public domain (USA) | 407 | 2,281 | 663 | 632 | 600 | 594 | 591 | 591 | 666 | 597 | 735 | 568 | 594 | 667 | 601 | 601 | 601 | 661 |
| JSON, pretty-printed (Wikidata Q2 Earth) | CC0 1.0 | 583 | 9,526 | 2,790 | 2,887 | 3,513 | 3,560 | 2,723 | 2,723 | 2,742 | 2,731 | 2,758 | 2,758 | 3,560 | 2,814 | 3,254 | 3,245 | 3,245 | 3,206 |
| JSON, minified (Wikidata Q2 Earth) | CC0 1.0 | 583 | 5,289 | 1,931 | 1,947 | 2,214 | 2,313 | 1,848 | 1,848 | 1,802 | 1,786 | 1,819 | 1,819 | 2,313 | 1,874 | 2,398 | 2,389 | 2,389 | 2,266 |
| JSON, pretty-printed (Wikidata Q90 Paris) | CC0 1.0 | 270 | 3,908 | 1,266 | 1,316 | 1,572 | 1,575 | 1,239 | 1,239 | 1,268 | 1,246 | 1,252 | 1,261 | 1,575 | 1,306 | 1,425 | 1,419 | 1,419 | 1,438 |
| JSON, minified (Wikidata Q90 Paris) | CC0 1.0 | 270 | 2,367 | 854 | 869 | 956 | 986 | 823 | 823 | 821 | 796 | 805 | 814 | 986 | 859 | 1,014 | 1,008 | 1,008 | 991 |
Tokens per word, tokens per character and ratios for each text are in the CSV.
Method
Parallel text: FLORES-200 dev and devtest (2,009 aligned sentences per language, CC BY-SA 4.0, tarball pinned by SHA-256) for Modern Standard, Egyptian, North Levantine, Najdi and Mesopotamian Arabic, Hindi, Urdu, Chinese, French and Spanish against English, each sentence counted on its own; Tatoeba sentence pairs (CC BY 2.0 FR) for Gulf Arabic and Arabizi. Ratio = pooled tokens ÷ pooled tokens of the English side, with a 95% interval from a seeded bootstrap over sentences (10,000 resamples). Other texts: Wikipedia lead sections (CC BY-SA 4.0), Project Gutenberg (public domain) and Wikidata JSON (CC0), stored with source URL, licence and revision. Tokens: the official tokenizer file at a pinned Hugging Face commit, encoded with @huggingface/tokenizers with no special tokens; o200k_base via gpt-tokenizer; Kimi's tiktoken rank file with a tiktoken-compatible counter. Gated Meta and Google files were read from ungated mirrors proven byte-identical to the official file. Every new count matched the Rust tokenizers library or Python tiktoken. Words: word-like segments from Intl.Segmenter. Characters: Unicode code points.
- Parallel text (FLORES-200): the dev (997) and devtest (1,012) splits, 2,009 sentences per language, aligned line by line across languages. The sentences come from English Wikinews, Wikijunior and Wikivoyage articles and were translated by professional translators for Meta's No Language Left Behind project. We use eng_Latn, arb_Arab, arz_Arab, apc_Arab, ars_Arab, acm_Arab, hin_Deva, urd_Arab, zho_Hans, fra_Latn, spa_Latn. Each sentence is encoded on its own and the counts are summed; the ratio against English is pooled tokens ÷ pooled English tokens, on the same sentences.
- Which FLORES-200 version: the original FLORES-200 release, downloaded from Meta's server (last modified 2022-07-14, 25,585,843 bytes, SHA-256
b8b0b76783024b85797e5cc75064eb83fc5288b41e9654dabc7be6ae944011f6), the download linked from the archived facebookresearch/flores repo at commita6c830c6e105. The tarball has no commit hash of its own, so the SHA-256 is the pin; the script refuses any other file. Its maintained successor, FLORES+ (checked at revision1147e2dc59f3), is gated behind a Hugging Face account and two click-through conditions, including not re-hosting the text where web crawlers can reach it. We did not use it, so FLORES+ corrections made after FLORES-200 are not reflected. - Arabic varieties: FLORES-200 labels them arb_Arab (Modern Standard Arabic), arz_Arab (Egyptian Arabic), apc_Arab (North Levantine Arabic: Syria and Lebanon), ars_Arab (Najdi Arabic, central Saudi Arabia) and acm_Arab (Mesopotamian Arabic, Iraq). None is Gulf Arabic (Khaleeji, ISO 639-3 afb). We compared each dialect sentence with the MSA sentence on the same line: 1,339 of 2,009 Najdi sentences are identical to MSA, against 2 arz_Arab, 1 apc_Arab, 41 acm_Arab. So the Najdi set is mostly MSA. Gulf Arabic keeps the 49 Tatoeba sentences tagged afb, and Arabizi keeps 39 Latin-script Tatoeba sentences from the Gulf, Egyptian and Levantine sets. 1 Tatoeba sentence was left out after review (reason in the JSON).
- Other texts: 4 from Tatoeba (CC BY 2.0 FR), 30 from Wikipedia (CC BY-SA 4.0), 3 from Project Gutenberg (Public domain (USA)), 4 from Wikidata (CC0 1.0). Wikipedia gives the lead section of the same three articles (Water, Moon, Internet) in eight language editions, plus code blocks from programming articles. Gutenberg gives the opening of three novels. Wikidata gives entity JSON, pretty-printed and minified. English tokens per 1,000 words, the site's headline figure, comes from the English Wikipedia and Gutenberg prose only.
- Uncertainty: for each tokenizer and parallel set, 10,000 bootstrap resamples of the sentence pairs (with replacement, the same pairs on both sides), seeded so a rerun gives the same numbers (seed 20261115); the interval is the 2.5th to 97.5th percentile of the pooled ratio. It covers sampling of sentences, not the choice of corpus: news-style FLORES-200 sentences and short everyday Tatoeba sentences give different ratios (see the change note below).
- Tokenizers: the official tokenizer file at the commits listed above; GPT uses OpenAI's o200k_base. Gemma 4 uses Google's base-model tokenizer, since BroadNet has not published one for its build. Counts exclude chat-template and special tokens, so a real request adds a few tokens per message.
- Gated files: Meta (Llama 3.3, Llama 4) and Google (Gemma 3) gate their repos. The script downloads the same file from an ungated mirror, computes its SHA-256 and size, rebuilds the Git LFS pointer, hashes it as a Git blob, and stops unless that ID equals the one the official repo lists for
tokenizer.jsonat the pinned commit. The Hub API lists that ID without an account, so anyone can repeat the check. - Kimi: Moonshot publishes no
tokenizer.json; the official tokenizer istiktoken.modelplus the split regex intokenization_kimi.py. The script checks that regex is unchanged and counts with a tiktoken-compatible encoder. - Checks: in version 3, every count for the 11 added tokenizers matched an independent reference on all texts: the Rust
tokenizerslibrary for the tokenizer.json files and Pythontiktokenfor Kimi. For version 4 the FLORES-200 counts were recomputed independently with the same Python libraries for every tokenizer and language, sentence by sentence, and matched; the texts kept from version 3 gave the same counts, words and characters as before. - Ratio against English: for parallel sets, tokens ÷ tokens of the English side. The CSV also gives every text's tokens per word ÷ English prose tokens per word (same tokenizer).
- Update frequency: token counts are re-measured when a tokenizer or the corpus changes; costs are recomputed on every build from BroadNet's pricing.json.
- Limits: FLORES-200 is translated text (mostly from English, so the English side is the original), in a formal news and encyclopedia register; dialect translations follow MSA closely in places, and Najdi almost entirely. Chat, support tickets and code-mixed text will differ. The Gulf Arabic and Arabizi sets are small. Treat ratios as indicative and test your own text.
Licensing: what you may reuse
Our token, word and character counts, ratios, intervals, costs, tables and method are CC BY 4.0. FLORES-200 sentences are CC BY-SA 4.0 (NLLB Team et al., No Language Left Behind: Scaling Human-Centered Machine Translation (2022), arXiv:2207.04672; FLORES-200 evaluation benchmark.). Our CC BY 4.0 files contain no FLORES-200 text: only our own token, word and character counts for each language, with the corpus identified by its pinned tarball (SHA-256 b8b0b76783024b85797e5cc75064eb83fc5288b41e9654dabc7be6ae944011f6) and splits (dev, devtest). Get the sentences from https://dl.fbaipublicfiles.com/nllb/flores200_dataset.tar.gz. Cite FLORES-200 as: NLLB Team et al., No Language Left Behind: Scaling Human-Centered Machine Translation (2022), arXiv:2207.04672; FLORES-200 evaluation benchmark. The Tatoeba, Wikipedia, Gutenberg and Wikidata texts in the JSON keep their own licences, given per text.
What changed in version 4
Version 4 replaces the short Tatoeba sentence sets (33 to 73 sentences, 450 to 1,517 characters per language) with the 2,009 aligned FLORES-200 sentences for every language FLORES-200 covers, and adds Najdi and Mesopotamian Arabic. Gulf Arabic and Arabizi keep their Tatoeba sets: FLORES-200 has no equivalent. Each ratio now has a 95% interval. The tokenizers, the English prose texts and the code and JSON texts are unchanged, so English tokens per 1,000 words (1,290) did not move. Ratios moved because the sample is 28 to 61 times as many sentences and in a different register, not because a version 3 count was wrong: version 3 measured what its sentences contained. Short everyday sentences gave lower Arabic and Hindi ratios than news-style FLORES-200 text. Old against new, five core tokenizers:
| Set | Sentences v3 → v4 | DeepSeek V4.1 Flash | GLM 5.3 | Qwen3.8 27B | Gemma 4 VL 26B | GPT (o200k) |
|---|---|---|---|---|---|---|
| Arabic MSA Tatoeba → FLORES-200 | 51 → 2,009 | 1.41× → 1.62× (1.61–1.63) | 1.53× → 1.77× (1.76–1.78) | 1.16× → 1.31× (1.30–1.32) | 1.22× → 1.47× (1.46–1.48) | 1.29× → 1.38× (1.37–1.39) |
| Gulf Arabic Tatoeba → Tatoeba | 49 → 49 | 1.44× → 1.44× (1.34–1.53) | 1.57× → 1.57× (1.47–1.68) | 1.21× → 1.21× (1.13–1.29) | 1.22× → 1.22× (1.12–1.32) | 1.32× → 1.32× (1.22–1.42) |
| Egyptian Arabic Tatoeba → FLORES-200 | 38 → 2,009 | 1.42× → 1.63× (1.61–1.64) | 1.53× → 1.75× (1.74–1.77) | 1.24× → 1.35× (1.34–1.36) | 1.21× → 1.44× (1.42–1.45) | 1.32× → 1.41× (1.40–1.42) |
| Levantine Arabic Tatoeba → FLORES-200 | 33 → 2,009 | 1.41× → 1.58× (1.57–1.59) | 1.50× → 1.70× (1.68–1.71) | 1.16× → 1.29× (1.28–1.30) | 1.09× → 1.41× (1.40–1.43) | 1.26× → 1.35× (1.34–1.36) |
| Najdi Arabic new → FLORES-200 | — → 2,009 | — → 1.63× (1.61–1.64) | — → 1.77× (1.76–1.79) | — → 1.31× (1.30–1.32) | — → 1.47× (1.46–1.49) | — → 1.39× (1.38–1.40) |
| Mesopotamian Arabic new → FLORES-200 | — → 2,009 | — → 1.62× (1.61–1.63) | — → 1.75× (1.74–1.77) | — → 1.32× (1.31–1.33) | — → 1.46× (1.45–1.47) | — → 1.39× (1.38–1.40) |
| Arabizi Tatoeba → Tatoeba | 39 → 39 | 1.50× → 1.50× (1.40–1.61) | 1.48× → 1.48× (1.38–1.59) | 1.38× → 1.38× (1.29–1.47) | 1.30× → 1.30× (1.21–1.39) | 1.45× → 1.45× (1.35–1.55) |
| French Tatoeba → FLORES-200 | 46 → 2,009 | 1.35× → 1.54× (1.53–1.55) | 1.28× → 1.50× (1.49–1.51) | 1.20× → 1.38× (1.37–1.39) | 1.19× → 1.40× (1.39–1.41) | 1.24× → 1.36× (1.35–1.37) |
| Spanish Tatoeba → FLORES-200 | 46 → 2,009 | 1.18× → 1.50× (1.48–1.51) | 1.10× → 1.44× (1.43–1.45) | 1.05× → 1.32× (1.31–1.33) | 0.96× → 1.29× (1.29–1.30) | 1.09× → 1.32× (1.31–1.33) |
| Hindi Tatoeba → FLORES-200 | 73 → 2,009 | 2.49× → 2.81× (2.79–2.83) | 3.67× → 4.33× (4.29–4.37) | 1.74× → 2.02× (2.00–2.03) | 1.21× → 1.31× (1.31–1.32) | 1.59× → 1.58× (1.56–1.59) |
| Urdu Tatoeba → FLORES-200 | 46 → 2,009 | 2.43× → 2.37× (2.36–2.39) | 2.82× → 2.75× (2.73–2.78) | 1.82× → 1.98× (1.97–2.00) | 1.24× → 1.53× (1.52–1.54) | 1.65× → 1.66× (1.65–1.67) |
| Chinese Tatoeba → FLORES-200 | 37 → 2,009 | 0.78× → 0.95× (0.94–0.96) | 0.81× → 0.96× (0.95–0.97) | 0.79× → 0.92× (0.91–0.93) | 0.89× → 1.09× (1.08–1.10) | 1.09× → 1.26× (1.25–1.27) |
All 16 tokenizers, old and new, are in the JSON (changes). Version 3 (measured 2026-11-15) is superseded by this page.
What changed in version 3
Version 3 adds 11 tokenizers used by widely hosted open-weight models: gpt-oss (o200k_harmony), Llama 3.3 70B, Llama 4 Maverick, Kimi K3, MiniMax M3, Gemma 3 27B, Phi-4, Mistral Small 3.1, Ministral 3 14B, Nemotron 3 Ultra, Qwen3 32B. The corpus and the five v2 tokenizers were unchanged, and so were the headline averages, which use the five core tokenizers. Version 2 replaced the 12 short texts of version 1 with 52 openly licensed texts; English prose went from 1,142 to 1,290 tokens per 1,000 words.
What do the CSV columns mean?
| Column | Meaning |
|---|---|
text_id | Text ID; the JSON "corpus" holds the full text, source, authors and sentence or revision IDs. |
text | Short label for the text (the text itself is only in the JSON). |
group | Language or text kind, e.g. ar-msa, arabizi, code-python, json. |
kind | parallel (FLORES-200 or Tatoeba sentences), parallel-en (their English side), prose, code or json. |
source | Where the text comes from: FLORES-200, Tatoeba, Wikipedia, Project Gutenberg or Wikidata. |
license | Licence of the source text (it is not relicensed). FLORES-200 text is not in our files at all; only our counts are. |
url | Source link: the pinned FLORES-200 page, first Tatoeba sentence ID, or the exact page revision. |
sentences | Parallel texts: number of sentences (FLORES-200: 997 dev + 1,012 devtest). |
chars | Unicode code points. |
words | Word-like segments from Intl.Segmenter. |
tokenizer | Tokenizer ID: deepseek, glm, qwen, gemma, gpt (o200k_base), gptoss (o200k_harmony), llama3, llama4, kimi, minimax, gemma3, phi4, mistral, ministral, nemotron or qwen3. The JSON "tokenizers" list gives each repo and pinned commit. |
core | yes for the five v2 tokenizers behind the site's headline averages and token tools; no for the 11 added in v3. |
tokens | Token count, no special or chat-template tokens. |
tokens_per_word | tokens ÷ words. |
tokens_per_char | tokens ÷ chars. |
vs_english_per_word | tokens_per_word ÷ English prose tokens per word, same tokenizer. |
vs_english_parallel | Parallel texts only: tokens ÷ tokens of the English translation of the same sentences. |
vs_english_parallel_ci95_low | Lower end of the 95% bootstrap interval for vs_english_parallel (10,000 seeded resamples of sentences). |
vs_english_parallel_ci95_high | Upper end of that interval. |
broadnet_model | BroadNet model that uses this tokenizer, if any. |
input_usd_per_1m_chars | tokens_per_char × that model's input price per 1M tokens (BroadNet pricing.json at build time). |
How to cite this dataset
Token counts, ratios, cost figures, tables and method: CC BY 4.0. Free to use, share and adapt with attribution and a link to this page. FLORES-200 text is not included and stays CC BY-SA 4.0; when you use the FLORES-200 ratios, also cite NLLB Team et al., No Language Left Behind: Scaling Human-Centered Machine Translation (2022), arXiv:2207.04672; FLORES-200 evaluation benchmark.
The source texts in the JSON download are not relicensed: each keeps its source licence, given per text. Wikipedia texts are CC BY-SA 4.0, so a copy or adaptation of those texts must stay under CC BY-SA 4.0 and credit Wikipedia contributors (the revision link lists them). Tatoeba sentences are CC BY 2.0 FR: credit the authors listed per text (each sentence ID links to its author). Wikidata JSON is CC0; the Project Gutenberg excerpts are public domain. FLORES-200 sentences (CC BY-SA 4.0, NLLB Team, Meta) are not in the download at all: only our counts are, with the corpus pinned and linked.
Plain text
BroadRouter (2026). Tokenizer efficiency across languages: 16 tokenizers from DeepSeek, GLM, Qwen, Gemma, OpenAI, Meta, Moonshot, MiniMax, Microsoft, Mistral and NVIDIA (version 4.0) [Data set]. BroadNet Technologies. https://broadrouter.com/data/tokenizer-efficiency/BibTeX
@misc{broadrouter_tokenizer_efficiency,
author = {BroadRouter},
title = {Tokenizer efficiency across languages: 16 tokenizers from DeepSeek, GLM, Qwen, Gemma, OpenAI, Meta, Moonshot, MiniMax, Microsoft, Mistral and NVIDIA},
year = {2026},
version = {4.0},
publisher = {BroadNet Technologies},
howpublished = {\url{https://broadrouter.com/data/tokenizer-efficiency/}},
note = {Data set, data as of 2026-11-15},
}Questions
How many tokens is 1,000 words?
About 1,290 tokens for English prose, averaged over the five core tokenizers (1,245 to 1,320 across all 16). Arabic, Hindi, code and JSON need more. Convert any amount with the tokens to words converter.
Which tokenizer is most efficient for Arabic?
For Modern Standard Arabic, Ministral 3 14B needed the fewest tokens relative to the same 2,009 English sentences (1.25×, 95% interval 1.24–1.25) and Phi-4 the most (3.03×, 3.01–3.05). Fewer tokens is not the same as cheaper: see what the same text costs by provider, or measure your own text with the Arabic token counter.
Which tokenizer is most efficient for Hindi and Urdu?
Gemma 4 VL 26B needed the fewest tokens for Hindi (1.31× the English count) and Phi-4 the most (4.78×). For Urdu the range is 1.53× (Gemma 4 VL 26B) to 4.36× (Phi-4).
Does Arabizi use more tokens than Arabic script?
Arabizi costs 1.30× (Gemma 4 VL 26B) to 1.55× (Kimi K3) the English token count. Latin letters reuse English word pieces, but the digits used for Arabic sounds (2, 3, 7) and non-standard spelling split words further.
Why does Arabic cost more tokens than English?
Tokenizers learn their vocabulary from training text. When a vocabulary holds fewer Arabic word pieces, Arabic words split into more tokens, and API bills are per token. Dialect spelling and Arabizi push the count higher.
How sure are these ratios?
Each ratio against English comes with a 95% interval from a bootstrap over sentences (10,000 resamples, fixed seed). On the 2,009 FLORES-200 sentences per language no interval is wider than 0.08 (ratio points). The Gulf Arabic and Arabizi sets have under 50 Tatoeba sentences each, and their intervals are up to 0.34 wide, so treat those two as rough.
Can I reproduce these numbers?
Yes. Download FLORES-200 from Meta (the SHA-256 is on this page), then in the site repository run node scripts/dataset-tokenizer-flores.mjs and node scripts/dataset-tokenizer-efficiency.mjs with FLORES_DIR set. The JSON download includes every other text with its source, licence and revision or sentence IDs, and every tokenizer with its repo and pinned commit. The script re-checks the gated Meta and Google tokenizer files against the official repos before counting.
Can I use this data?
Yes. The token counts, ratios, intervals and costs are CC BY 4.0; cite them as shown under "How to cite". FLORES-200 sentences are CC BY-SA 4.0 and are not in our files; we publish only our counts, with the corpus pinned and linked. The texts in the JSON keep their own licences: Wikipedia CC BY-SA 4.0 (share-alike), Tatoeba CC BY 2.0 FR (credit the authors listed per text), Wikidata CC0 and the Gutenberg excerpts public domain.
Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity
Sources
- DeepSeek V4.1 Flash: tokenizer.json @ 314113f
- GLM 5.3: tokenizer.json @ ba32f8c
- Qwen3.8 27B: tokenizer.json @ 3756181
- Gemma 4 VL 26B: tokenizer.json @ 4d7ae49
- gpt-oss (o200k_harmony): tokenizer.json @ b5c939d
- Llama 3.3 70B: tokenizer.json @ 6f6073b (read via unsloth/Llama-3.3-70B-Instruct, byte-identical)
- Llama 4 Maverick: tokenizer.json @ 73d1471 (read via unsloth/Llama-4-Maverick-17B-128E-Instruct, byte-identical)
- Kimi K3: tiktoken.model @ f831ab6
- MiniMax M3: tokenizer.json @ d95e30c
- Gemma 3 27B: tokenizer.json @ 005ad34 (read via unsloth/gemma-3-27b-it, byte-identical)
- Phi-4: tokenizer.json @ 2db69c1
- Mistral Small 3.1: tokenizer.json @ 68faf51
- Ministral 3 14B: tokenizer.json @ 29439f8
- Nemotron 3 Ultra: tokenizer.json @ 77df655
- Qwen3 32B: tokenizer.json @ 9216db5
- gpt-tokenizer (o200k_base)
- FLORES-200 (CC BY-SA 4.0): dev + devtest, tarball SHA-256 b8b0b7678302…, linked from facebookresearch/flores @ a6c830c
- NLLB Team, No Language Left Behind (FLORES-200 paper), arXiv 2207.04672
- FLORES-200 (CC BY-SA 4.0 (counts only; text not reproduced)), 11 texts
- Tatoeba (CC BY 2.0 FR), 4 texts
- Wikipedia (CC BY-SA 4.0), 30 texts
- Project Gutenberg (Public domain (USA)), 3 texts
- Wikidata (CC0 1.0), 4 texts