BroadRouter

Tokenizer efficiency across languages

Updated

1,000 words of English prose is about 1,290 tokens. The same 2,009 FLORES-200 sentences in Modern Standard Arabic need 1.25× to 3.03× as many tokens as in English, depending on the tokenizer. We measured 16 official tokenizers pinned to exact commits; counts, 95% intervals and method are free to download.

Download CSV Download JSON (with texts) Croissant metadata Counts CC BY 4.0, texts under their source licences, no FLORES-200 text · version 4.0 · measured 2026-11-15

Key findings

  1. 1,000 words of English prose is about 1,290 tokens, averaged over the five core tokenizers, or about 4.6 characters per token. Across all 16 tokenizers it ranges from 1,245 (MiniMax M3) to 1,320 (Qwen3.8 27B).
  2. Modern Standard Arabic needs 1.25× (Ministral 3 14B, 95% interval 1.24–1.25) to 3.03× (Phi-4, 3.01–3.05) the tokens of the same 2,009 FLORES-200 sentences in English. The dearest tokenizer needs 2.37× the tokens of the cheapest for the same Arabic.
  3. Arabic dialects in Arabic script: Egyptian Arabic 1.29× (Mistral Small 3.1) to 2.95× (Phi-4); Levantine Arabic 1.24× (Mistral Small 3.1) to 2.82× (Phi-4); Mesopotamian Arabic 1.26× (Ministral 3 14B) to 2.98× (Phi-4); Najdi Arabic 1.25× (Ministral 3 14B) to 3.03× (Phi-4); Gulf Arabic 1.15× (Ministral 3 14B) to 2.54× (Phi-4) the English token count. In FLORES-200, 1,339 of the 2,009 Najdi sentences (67%) are identical to the Modern Standard Arabic ones, so the Najdi figures are close to MSA by construction. Gulf Arabic has no FLORES-200 set; its 49 Tatoeba sentences give wide intervals (for example 1.13–1.29 on Qwen3.8).
  4. Arabizi, Arabic dialect typed in Latin letters and digits, costs 1.30× (Gemma 4 VL 26B) to 1.55× (Kimi K3) the English token count. The gap between tokenizers is narrower than for dialect in Arabic script (1.11× between dearest and cheapest, against 2.24×, 2.23×, 2.31×, 2.37×, 2.14×).
  5. Hindi needs 1.31× (Gemma 4 VL 26B, 95% interval 1.31–1.32) to 4.78× (Phi-4, 4.74–4.82) the English token count, and Urdu 1.53× (Gemma 4 VL 26B) to 4.36× (Phi-4). Hindi shows the widest gap between tokenizers: its dearest tokenizer needs 3.63× the tokens of its cheapest.
  6. Chinese needs 0.88× (Kimi K3) to 1.88× (Phi-4) the tokens of the same English sentences: tokenizers trained heavily on Chinese need fewer tokens for Chinese than for English.
  7. Code needs more tokens per character than prose: Python 3.7, JavaScript 3.7, SQL 3.8 characters per token on the five core tokenizers, against 4.6 for English prose. JSON ranges from 3,845 to 4,944 tokens per 1,000 words, and pretty-printed JSON uses 1.53× the tokens of the same JSON minified.
  8. Gemma 4 VL 26B and Gemma 3 27B, GPT (o200k) and gpt-oss (o200k_harmony), Ministral 3 14B and Nemotron 3 Ultra gave exactly the same count on all 52 texts, so in practice they share one vocabulary.
  9. On BroadNet's prices (2026-11-14), 1M characters of Arabic (MSA) input costs $0.0343 on BroadNet Gemma 4 VL 26B and $0.322 on GLM 5.3; English costs $0.0219 and $0.169. Every provider's price, by language, is in the real cost by language dataset.

How many more tokens does each language need than English?

Tokens for a set of sentences ÷ tokens for their English translations (same meaning). 1.00× means the language costs the same as English. 10 sets use the same 2,009 FLORES-200 sentences; Gulf Arabic and Arabizi use Tatoeba sentence pairs, because FLORES-200 has no Gulf Arabic or Arabizi. Under each ratio: its 95% bootstrap interval. One row per tokenizer; on a phone each tokenizer is a card.

TokenizerArabic MSAGulf Arabic (Tatoeba)Egyptian ArabicLevantine ArabicNajdi ArabicMesopotamian ArabicArabizi (Tatoeba)FrenchSpanishHindiUrduChinese
DeepSeek V4.1 Flash1.62× (1.61–1.63)1.44× (1.34–1.53)1.63× (1.61–1.64)1.58× (1.57–1.59)1.63× (1.61–1.64)1.62× (1.61–1.63)1.50× (1.40–1.61)1.54× (1.53–1.55)1.50× (1.48–1.51)2.81× (2.79–2.83)2.37× (2.36–2.39)0.95× (0.94–0.96)
GLM 5.31.77× (1.76–1.78)1.57× (1.47–1.68)1.75× (1.74–1.77)1.70× (1.68–1.71)1.77× (1.76–1.79)1.75× (1.74–1.77)1.48× (1.38–1.59)1.50× (1.49–1.51)1.44× (1.43–1.45)4.33× (4.29–4.37)2.75× (2.73–2.78)0.96× (0.95–0.97)
Qwen3.8 27B1.31× (1.30–1.32)1.21× (1.13–1.29)1.35× (1.34–1.36)1.29× (1.28–1.30)1.31× (1.30–1.32)1.32× (1.31–1.33)1.38× (1.29–1.47)1.38× (1.37–1.39)1.32× (1.31–1.33)2.02× (2.00–2.03)1.98× (1.97–2.00)0.92× (0.91–0.93)
Gemma 4 VL 26B1.47× (1.46–1.48)1.22× (1.12–1.32)1.44× (1.42–1.45)1.41× (1.40–1.43)1.47× (1.46–1.49)1.46× (1.45–1.47)1.30× (1.21–1.39)1.40× (1.39–1.41)1.29× (1.29–1.30)1.31× (1.31–1.32)1.53× (1.52–1.54)1.09× (1.08–1.10)
GPT (o200k)1.38× (1.37–1.39)1.32× (1.22–1.42)1.41× (1.40–1.42)1.35× (1.34–1.36)1.39× (1.38–1.40)1.39× (1.38–1.40)1.45× (1.35–1.55)1.36× (1.35–1.37)1.32× (1.31–1.33)1.58× (1.56–1.59)1.66× (1.65–1.67)1.26× (1.25–1.27)
gpt-oss (o200k_harmony)1.38× (1.37–1.39)1.32× (1.22–1.42)1.41× (1.40–1.42)1.35× (1.34–1.36)1.39× (1.38–1.40)1.39× (1.38–1.40)1.45× (1.35–1.55)1.36× (1.35–1.37)1.32× (1.31–1.33)1.58× (1.56–1.59)1.66× (1.65–1.67)1.26× (1.25–1.27)
Llama 3.3 70B1.66× (1.64–1.67)1.50× (1.40–1.60)1.65× (1.64–1.66)1.59× (1.58–1.60)1.66× (1.65–1.67)1.65× (1.64–1.66)1.49× (1.39–1.59)1.59× (1.58–1.60)1.54× (1.53–1.55)2.53× (2.51–2.55)3.09× (3.06–3.11)1.29× (1.28–1.30)
Llama 4 Maverick1.59× (1.58–1.60)1.45× (1.35–1.55)1.61× (1.59–1.62)1.55× (1.54–1.57)1.60× (1.58–1.61)1.59× (1.58–1.60)1.46× (1.36–1.55)1.36× (1.35–1.37)1.30× (1.29–1.31)1.65× (1.64–1.66)1.89× (1.88–1.90)1.07× (1.06–1.08)
Kimi K32.05× (2.04–2.07)1.86× (1.75–1.97)2.02× (2.01–2.04)1.94× (1.92–1.95)2.06× (2.04–2.07)2.03× (2.02–2.05)1.55× (1.44–1.66)1.72× (1.71–1.74)1.69× (1.68–1.71)2.75× (2.73–2.77)2.17× (2.16–2.19)0.88× (0.87–0.89)
MiniMax M31.36× (1.35–1.37)1.32× (1.23–1.43)1.40× (1.39–1.41)1.33× (1.32–1.34)1.36× (1.35–1.37)1.37× (1.36–1.38)1.51× (1.40–1.62)1.29× (1.28–1.30)1.25× (1.24–1.26)2.94× (2.92–2.96)3.78× (3.74–3.81)0.92× (0.91–0.93)
Gemma 3 27B1.47× (1.46–1.48)1.22× (1.12–1.32)1.44× (1.42–1.45)1.41× (1.40–1.43)1.47× (1.46–1.49)1.46× (1.45–1.47)1.30× (1.21–1.39)1.40× (1.39–1.41)1.29× (1.29–1.30)1.31× (1.31–1.32)1.53× (1.52–1.54)1.09× (1.08–1.10)
Phi-43.03× (3.01–3.05)2.54× (2.37–2.71)2.95× (2.93–2.98)2.82× (2.80–2.84)3.03× (3.01–3.06)2.98× (2.96–3.00)1.51× (1.40–1.61)1.60× (1.58–1.61)1.55× (1.53–1.56)4.78× (4.74–4.82)4.36× (4.33–4.40)1.88× (1.86–1.90)
Mistral Small 3.11.27× (1.26–1.28)1.17× (1.09–1.25)1.29× (1.28–1.30)1.24× (1.23–1.25)1.27× (1.26–1.28)1.27× (1.26–1.28)1.44× (1.33–1.55)1.31× (1.30–1.32)1.32× (1.31–1.33)2.76× (2.74–2.79)1.76× (1.75–1.78)1.42× (1.40–1.43)
Ministral 3 14B1.25× (1.24–1.25)1.15× (1.08–1.23)1.29× (1.28–1.30)1.24× (1.23–1.24)1.25× (1.24–1.26)1.26× (1.25–1.27)1.44× (1.33–1.55)1.31× (1.30–1.32)1.32× (1.31–1.33)1.80× (1.78–1.81)1.76× (1.75–1.77)1.42× (1.40–1.43)
Nemotron 3 Ultra1.25× (1.24–1.25)1.15× (1.08–1.23)1.29× (1.28–1.30)1.24× (1.23–1.24)1.25× (1.24–1.26)1.26× (1.25–1.27)1.44× (1.33–1.55)1.31× (1.30–1.32)1.32× (1.31–1.33)1.80× (1.78–1.81)1.76× (1.75–1.77)1.42× (1.40–1.43)
Qwen3 32B1.63× (1.62–1.64)1.47× (1.36–1.59)1.64× (1.62–1.65)1.58× (1.57–1.59)1.63× (1.62–1.65)1.62× (1.61–1.64)1.50× (1.39–1.60)1.57× (1.56–1.58)1.52× (1.51–1.53)4.43× (4.39–4.47)3.16× (3.14–3.19)1.01× (1.00–1.02)

Sentences per set: Arabic MSA 2,009 (FLORES-200), Gulf Arabic 49 (Tatoeba), Egyptian Arabic 2,009 (FLORES-200), Levantine Arabic 2,009 (FLORES-200), Najdi Arabic 2,009 (FLORES-200), Mesopotamian Arabic 2,009 (FLORES-200), Arabizi 39 (Tatoeba), French 2,009 (FLORES-200), Spanish 2,009 (FLORES-200), Hindi 2,009 (FLORES-200), Urdu 2,009 (FLORES-200), Chinese 2,009 (FLORES-200). Interval: percentile bootstrap over sentence pairs, seeded mulberry32 PRNG, 10,000 resamples, seed 20261115.

How many tokens is 1,000 words in each language?

All texts of a kind pooled: total tokens ÷ total words × 1,000. English is the English prose texts (Wikipedia and Project Gutenberg), not FLORES-200; the other languages pool their FLORES-200 sentences with their Wikipedia leads, so FLORES-200 text dominates them. Words are word-like segments from Intl.Segmenter; for Chinese these are segmented words, not characters. Characters per token for every tokenizer and text kind are in the CSV and JSON.

TokenizerEnglishArabic MSAEgyptian ArabicLevantine ArabicNajdi ArabicMesopotamian ArabicGulf ArabicArabiziFrenchSpanishHindiUrduChinese
DeepSeek V4.1 Flash1,2742,2212,2512,3702,2252,3212,3152,2761,6351,5482,9342,3021,124
GLM 5.31,2902,4392,4372,5622,4412,5312,5072,2481,6041,5004,5472,6831,138
Qwen3.8 27B1,3201,8301,8971,9721,8321,9302,2122,3361,4991,3942,1491,9601,109
Gemma 4 VL 26B1,2992,0271,9972,1372,0292,1072,2562,2761,4941,3471,3801,4881,292
GPT (o200k)1,2691,8851,9372,0191,8871,9772,0492,0981,4361,3551,6371,5991,478
gpt-oss (o200k_harmony)1,2691,8851,9372,0191,8871,9772,0492,0981,4361,3551,6371,5991,478
Llama 3.3 70B1,2892,2832,2912,4052,2872,3762,3942,2481,6981,6042,6533,0061,534
Llama 4 Maverick1,2642,1862,2232,3372,1862,2832,2812,1311,4441,3431,7261,8331,269
Kimi K31,2922,8302,8072,9212,8292,9272,9112,2621,8391,7602,8842,1141,046
MiniMax M31,2451,8281,9081,9691,8311,9302,0542,1961,3501,2743,0163,5991,069
Gemma 3 27B1,2992,0271,9972,1372,0292,1072,2562,2761,4941,3471,3801,4881,292
Phi-41,2894,1774,1034,2624,1744,2974,0492,2801,7051,6075,0134,2532,232
Mistral Small 3.11,3021,7901,8381,9231,7921,8791,9162,1921,4371,4042,9741,7621,727
Ministral 3 14B1,3021,7601,8361,9131,7631,8641,8922,1921,4371,4041,9341,7581,727
Nemotron 3 Ultra1,3021,7601,8361,9131,7631,8641,8922,1921,4371,4041,9341,7581,727
Qwen3 32B1,3062,2852,3122,4272,2852,3802,3502,2711,7051,6114,7293,1361,227

Code and JSON

TokenizerPython tokens / 1k wordsJavaScript tokens / 1k wordsSQL tokens / 1k wordsJSON tokens / 1k wordsPython chars / tokenJavaScript chars / tokenSQL chars / tokenJSON chars / token
DeepSeek V4.1 Flash2,1071,7962,2324,0103.763.763.853.09
GLM 5.32,0041,7882,0984,1143.953.794.103.00
Qwen3.8 27B2,1541,9372,3474,8393.683.483.662.56
Gemma 4 VL 26B2,3391,9862,4484,9443.383.403.512.50
GPT (o200k)2,0251,7552,1153,8883.913.854.073.17
gpt-oss (o200k_harmony)2,0251,7552,1153,8883.913.854.073.17
Llama 3.3 70B2,0041,7882,0983,8883.953.794.103.17
Llama 4 Maverick2,0251,7552,1513,8453.913.853.983.22
Kimi K32,0501,7902,1763,8893.863.773.943.17
MiniMax M32,0611,7572,1763,8993.833.853.943.17
Gemma 3 27B2,3391,9862,4484,9443.383.403.512.50
Phi-42,0041,7882,0984,0173.953.794.103.08
Mistral Small 3.12,0961,8202,3814,7433.773.723.612.60
Ministral 3 14B2,0961,7882,3814,7253.773.793.612.62
Nemotron 3 Ultra2,0961,7882,3814,7253.773.793.612.62
Qwen3 32B2,0181,8122,1854,6313.923.733.922.67

Which tokenizers are measured?

Each is the tokenizer file the model maker publishes, pinned to a commit. The first five (DeepSeek, GLM, Qwen3.8, Gemma 4 and GPT o200k) are the v2 core set behind the site's token tools; the other 11 were added in version 3. Which priced models use each tokenizer is checked file by file in the real cost by language dataset.

TokenizerMakerVocabularyFile and commitNote
DeepSeek V4.1 Flash · coreDeepSeek128,000deepseek-ai/DeepSeek-V4.1-Flash @ 314113fOfficial tokenizer.json from the DeepSeek-V4.1-Flash model repo.
GLM 5.3 · coreZ.ai154,820zai-org/GLM-5.3 @ ba32f8cOfficial tokenizer.json from the zai-org/GLM-5.3 model repo.
Qwen3.8 27B · coreQwen248,044Qwen/Qwen3.8-27B @ 3756181Official tokenizer.json from the Qwen/Qwen3.8-27B model repo.
Gemma 4 VL 26B · coreGoogle262,144google/gemma-4-26B-A4B-it @ 4d7ae49Google's Gemma 4 26B A4B tokenizer. BroadNet's build is based on this model; we assume it keeps the base tokenizer.
GPT (o200k) · coreOpenAI199,998o200k_base (gpt-tokenizer)o200k_base, the encoding used by GPT-4o, GPT-4.1, o-series and GPT-5 models (via gpt-tokenizer). Reference only: not sold by BroadNet.
gpt-oss (o200k_harmony)OpenAI199,998openai/gpt-oss-120b @ b5c939do200k_harmony is o200k_base plus the Harmony chat-format special tokens, so ordinary text should count the same as GPT (o200k).
Llama 3.3 70BMeta128,000meta-llama/Llama-3.3-70B-Instruct @ 6f6073bMeta's repo is gated. Bytes from the unsloth mirror, proven identical to Meta's tokenizer.json at the pinned commit. Proof: SHA-256 6b9e4e7fb171… rebuilds the LFS pointer with Git object 1c1d8d5c9024, the ID meta-llama/Llama-3.3-70B-Instruct lists for tokenizer.json.
Llama 4 MaverickMeta200,000meta-llama/Llama-4-Maverick-17B-128E-Instruct @ 73d1471Meta's repo is gated. Bytes from the unsloth mirror, proven identical to Meta's tokenizer.json at the pinned commit. Proof: SHA-256 172c9eb4beaf… rebuilds the LFS pointer with Git object b1fde397c877, the ID meta-llama/Llama-4-Maverick-17B-128E-Instruct lists for tokenizer.json.
Kimi K3Moonshot AI163,584moonshotai/Kimi-K3 @ f831ab6Moonshot publishes no tokenizer.json: the official tokenizer is the tiktoken rank file plus the regex in tokenization_kimi.py. Encoded with a tiktoken-compatible counter that matched Python tiktoken on every corpus text.
MiniMax M3MiniMax200,000MiniMaxAI/MiniMax-M3 @ d95e30cOfficial tokenizer.json from the MiniMaxAI/MiniMax-M3 model repo.
Gemma 3 27BGoogle262,144google/gemma-3-27b-it @ 005ad34Google's repo is gated. Bytes from the unsloth mirror, proven identical to Google's tokenizer.json at the pinned commit. Proof: SHA-256 4667f2089529… rebuilds the LFS pointer with Git object 29401f984828, the ID google/gemma-3-27b-it lists for tokenizer.json.
Phi-4Microsoft100,352microsoft/phi-4 @ 2db69c1Official tokenizer.json from microsoft/phi-4. Phi-4-mini and Phi-4-reasoning ship different tokenizer files and are not mapped to it.
Mistral Small 3.1Mistral AI131,072mistralai/Mistral-Small-3.1-24B-Instruct-2503 @ 68faf51Official tokenizer.json from the Mistral Small 3.1 repo.
Ministral 3 14BMistral AI131,072mistralai/Ministral-3-14B-Instruct-2512 @ 29439f8Official tokenizer.json from the Ministral 3 14B repo; the 3B and 8B repos ship the same file.
Nemotron 3 UltraNVIDIA131,072nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 @ 77df655Official tokenizer.json from NVIDIA's Nemotron 3 Ultra repo.
Qwen3 32BQwen151,643Qwen/Qwen3-32B @ 9216db5The original Qwen3 tokenizer. Qwen3.5 and later use the larger Qwen3.8 vocabulary, measured separately.

What does 1M characters cost on each BroadNet model?

Input cost of one million characters of each kind of text = tokens per character × the model's input price. Prices: BroadNet Inference rate card, 2026-11-14 (DeepSeek V4.1 Flash $0.175, GLM 5.3 $0.78, Qwen3.8 27B $0.175, BroadNet Gemma 4 VL 26B $0.10 per 1M input tokens). They update with the rate card. For every provider that sells a model with a measured tokenizer, see real cost by language.

Text kindDeepSeek V4.1 FlashGLM 5.3Qwen3.8 27BBroadNet Gemma 4 VL 26B
English prose$0.0375$0.169$0.0388$0.0219
English, FLORES-200 sentences$0.0359$0.161$0.0365$0.0206
Arabic (Modern Standard)$0.0658$0.322$0.0542$0.0343
Gulf Arabic$0.0725$0.350$0.0693$0.0404
Egyptian Arabic$0.0677$0.327$0.0570$0.0343
Levantine Arabic (North)$0.0689$0.332$0.0573$0.0355
Najdi Arabic (central Saudi Arabia)$0.0660$0.323$0.0543$0.0344
Mesopotamian Arabic (Iraq)$0.0673$0.327$0.0559$0.0349
Arabizi (Arabic in Latin letters)$0.0736$0.324$0.0756$0.0421
French$0.0463$0.203$0.0425$0.0242
Spanish$0.0451$0.195$0.0406$0.0224
Hindi$0.101$0.698$0.0740$0.0272
Urdu$0.0860$0.447$0.0732$0.0318
Chinese (Simplified)$0.103$0.463$0.101$0.0674
Code: Python$0.0466$0.198$0.0477$0.0296
Code: JavaScript$0.0465$0.206$0.0501$0.0294
Code: SQL$0.0455$0.191$0.0479$0.0285
JSON$0.0568$0.260$0.0685$0.0400

A cheaper per-token price can still cost more per character if the tokenizer splits that language finely. Output prices per 1M characters are in the JSON download.

Every text in the dataset

Raw token counts for all 52 texts and 16 tokenizers (wide table)
TextLicenceWordsCharsDeepSeek V4.1 FlashGLM 5.3Qwen3.8 27BGemma 4 VL 26BGPT (o200k)gpt-oss (o200k_harmony)Llama 3.3 70BLlama 4 MaverickKimi K3MiniMax M3Gemma 3 27BPhi-4Mistral Small 3.1Ministral 3 14BNemotron 3 UltraQwen3 32B
English, FLORES-200 sentencesCC BY-SA 4.043,194257,16052,73653,00453,69852,99552,40952,40952,96352,76052,92151,82152,99552,98354,30554,30554,30553,876
Modern Standard Arabic, FLORES-200 sentencesCC BY-SA 4.038,453226,91385,36393,70870,31477,91072,43772,43787,72183,999108,72870,24277,910160,52768,77667,64667,64687,789
Egyptian Arabic, FLORES-200 sentencesCC BY-SA 4.038,106221,93485,79992,87972,23776,08073,78673,78687,32284,743107,04172,69176,080156,49270,00369,91969,91988,118
North Levantine Arabic, FLORES-200 sentencesCC BY-SA 4.035,084211,22683,13689,88869,19874,98070,82770,82784,37182,001102,48769,06574,980149,54367,45967,11267,11285,152
Najdi Arabic, FLORES-200 sentencesCC BY-SA 4.038,508227,15285,69994,00070,52978,11672,67272,67288,06284,193108,92570,49978,116160,74369,02467,90467,90487,999
Mesopotamian Arabic, FLORES-200 sentencesCC BY-SA 4.036,747221,97185,30692,99470,90977,41872,63872,63887,31383,898107,54370,91277,418157,92069,03168,51068,51087,448
Hindi, FLORES-200 sentencesCC BY-SA 4.050,520256,462148,122229,555108,37169,62982,56982,569133,93587,034145,630152,28069,629253,044150,10897,51097,510238,687
Urdu, FLORES-200 sentencesCC BY-SA 4.054,365254,550125,216145,930106,56780,89186,94286,942163,44799,684114,949195,67680,891231,23995,77695,59795,597170,501
Chinese (Simplified), FLORES-200 sentencesCC BY-SA 4.044,51785,43550,08550,71249,39457,54565,87565,87568,36156,57446,59447,68257,54599,57076,89076,89076,89054,612
French, FLORES-200 sentencesCC BY-SA 4.049,654306,47981,06979,50774,32974,10171,18371,18384,20071,56491,25566,93074,10184,55871,24071,24071,24084,549
Spanish, FLORES-200 sentencesCC BY-SA 4.050,973306,27478,87176,41670,98868,62568,99968,99981,74768,41489,69064,85068,62581,90571,49571,49571,49582,085
Gulf Arabic, Tatoeba sentencesCC BY 2.0 FR2031,134470509449458416416486463591417458822389384384477
English side of Gulf Arabic, Tatoeba sentencesCC BY 2.0 FR2601,414327324372376316316324320318315376324333333333324
Arabizi (Arabic dialects in Latin letters), Tatoeba sentencesCC BY 2.0 FR2141,158487481500487449449481456484470487488469469469486
English side of Arabizi (Arabic dialects in Latin letters), Tatoeba sentencesCC BY 2.0 FR2601,294324324363375310310323313312311375324325325325325
English Wikipedia: WaterCC BY-SA 4.02741,706351364370356355355364348362346356364361361361368
Arabic (Modern Standard) Wikipedia: ماءCC BY-SA 4.02221,346507588435473443443534508667433473944425419419547
Egyptian Arabic Wikipedia: ميهCC BY-SA 4.078419194196180180178178194185221177180290175175175203
French Wikipedia: EauCC BY-SA 4.02491,620461458434409429429483418504385409484412412412484
Spanish Wikipedia: AguaCC BY-SA 4.02051,229335338321300305305364296390285300365316316316370
Hindi Wikipedia: जलCC BY-SA 4.03211,5759101,4087174935695698266038889294931,5439086526521,460
Chinese Wikipedia: 水CC BY-SA 4.0276505288302297355420420419323275275355584482482482334
Urdu Wikipedia: پانیCC BY-SA 4.0103489246295231198196196343216235394198451215215215346
English Wikipedia: MoonCC BY-SA 4.03652,149464472484479466466471462469458479471484484484485
Arabic (Modern Standard) Wikipedia: القمرCC BY-SA 4.02511,4605526214554934724725815537154604931,052453437437584
Egyptian Arabic Wikipedia: قمرCC BY-SA 4.02761,4805696475355405425425935837055275401,024508508508626
French Wikipedia: LuneCC BY-SA 4.03422,087545557506498498498577488611459498577491491491585
Spanish Wikipedia: LunaCC BY-SA 4.02431,372348355320307309309368305394289307368322322322368
Hindi Wikipedia: चन्द्रमाCC BY-SA 4.03061,5428541,3346874255075078115558288514251,4839066266261,413
Chinese Wikipedia: 月球CC BY-SA 4.06051,0876686756828038778779117556246258031,2441,0271,0271,027772
Urdu Wikipedia: چاندCC BY-SA 4.02641,1625336694693503793797814365109313501,112415412412798
English Wikipedia: InternetCC BY-SA 4.01891,338234233234235233233233233233232235233235235235233
Arabic (Modern Standard) Wikipedia: إنترنتCC BY-SA 4.0111767268313220244241241281277355226244534223215215288
Egyptian Arabic Wikipedia: انترنتCC BY-SA 4.087551189203156162160160195187238162162353161161161191
French Wikipedia: InternetCC BY-SA 4.01821,228315304293292279279327278350265292327294294294336
Spanish Wikipedia: InternetCC BY-SA 4.01941,254323300283272271271318263350260272318300300300317
Hindi Wikipedia: अंतरजालCC BY-SA 4.03392,0121,1861,8268785186226221,0336581,1561,2205182,0401,1738038031,899
Chinese Wikipedia: 互联网CC BY-SA 4.0251471250253257293317317333272236236293493438438438280
Urdu Wikipedia: انٹرنیٹCC BY-SA 4.0143775314362275210237237389268325472210606259256256444
Python code (Wikipedia: Python syntax and semantics)CC BY-SA 4.02081,706452427453501427427427427434436501427446446446431
Python code (Wikipedia: Python (programming language))CC BY-SA 4.072508138134150154140140134140140141154134141141141134
JavaScript code (Wikipedia: JavaScript syntax)CC BY-SA 4.02381,513436434461470426426434426434427470434439431431436
JavaScript code (Wikipedia: JavaScript)CC BY-SA 4.02521,799444442488503434434442434443434503442453445445452
SQL code (Wikipedia: Select (SQL))CC BY-SA 4.02121,509460436475497439439436437440442497436476476476456
SQL code (Wikipedia: Join (SQL))CC BY-SA 4.01451,555337313363377316316313331337335377313374374374324
English novel: Pride and Prejudice (Austen, 1813)Public domain (USA)3551,841458457473467447447457452465437467457460460460459
French novel: Candide (Voltaire, 1759)Public domain (USA)3181,908565547518515503503565509618474515565488488488563
Spanish novel: Don Quijote (Cervantes, 1605)Public domain (USA)4072,281663632600594591591666597735568594667601601601661
JSON, pretty-printed (Wikidata Q2 Earth)CC0 1.05839,5262,7902,8873,5133,5602,7232,7232,7422,7312,7582,7583,5602,8143,2543,2453,2453,206
JSON, minified (Wikidata Q2 Earth)CC0 1.05835,2891,9311,9472,2142,3131,8481,8481,8021,7861,8191,8192,3131,8742,3982,3892,3892,266
JSON, pretty-printed (Wikidata Q90 Paris)CC0 1.02703,9081,2661,3161,5721,5751,2391,2391,2681,2461,2521,2611,5751,3061,4251,4191,4191,438
JSON, minified (Wikidata Q90 Paris)CC0 1.02702,3678548699569868238238217968058149868591,0141,0081,008991

Tokens per word, tokens per character and ratios for each text are in the CSV.

Method

Parallel text: FLORES-200 dev and devtest (2,009 aligned sentences per language, CC BY-SA 4.0, tarball pinned by SHA-256) for Modern Standard, Egyptian, North Levantine, Najdi and Mesopotamian Arabic, Hindi, Urdu, Chinese, French and Spanish against English, each sentence counted on its own; Tatoeba sentence pairs (CC BY 2.0 FR) for Gulf Arabic and Arabizi. Ratio = pooled tokens ÷ pooled tokens of the English side, with a 95% interval from a seeded bootstrap over sentences (10,000 resamples). Other texts: Wikipedia lead sections (CC BY-SA 4.0), Project Gutenberg (public domain) and Wikidata JSON (CC0), stored with source URL, licence and revision. Tokens: the official tokenizer file at a pinned Hugging Face commit, encoded with @huggingface/tokenizers with no special tokens; o200k_base via gpt-tokenizer; Kimi's tiktoken rank file with a tiktoken-compatible counter. Gated Meta and Google files were read from ungated mirrors proven byte-identical to the official file. Every new count matched the Rust tokenizers library or Python tiktoken. Words: word-like segments from Intl.Segmenter. Characters: Unicode code points.

  • Parallel text (FLORES-200): the dev (997) and devtest (1,012) splits, 2,009 sentences per language, aligned line by line across languages. The sentences come from English Wikinews, Wikijunior and Wikivoyage articles and were translated by professional translators for Meta's No Language Left Behind project. We use eng_Latn, arb_Arab, arz_Arab, apc_Arab, ars_Arab, acm_Arab, hin_Deva, urd_Arab, zho_Hans, fra_Latn, spa_Latn. Each sentence is encoded on its own and the counts are summed; the ratio against English is pooled tokens ÷ pooled English tokens, on the same sentences.
  • Which FLORES-200 version: the original FLORES-200 release, downloaded from Meta's server (last modified 2022-07-14, 25,585,843 bytes, SHA-256 b8b0b76783024b85797e5cc75064eb83fc5288b41e9654dabc7be6ae944011f6), the download linked from the archived facebookresearch/flores repo at commit a6c830c6e105. The tarball has no commit hash of its own, so the SHA-256 is the pin; the script refuses any other file. Its maintained successor, FLORES+ (checked at revision 1147e2dc59f3), is gated behind a Hugging Face account and two click-through conditions, including not re-hosting the text where web crawlers can reach it. We did not use it, so FLORES+ corrections made after FLORES-200 are not reflected.
  • Arabic varieties: FLORES-200 labels them arb_Arab (Modern Standard Arabic), arz_Arab (Egyptian Arabic), apc_Arab (North Levantine Arabic: Syria and Lebanon), ars_Arab (Najdi Arabic, central Saudi Arabia) and acm_Arab (Mesopotamian Arabic, Iraq). None is Gulf Arabic (Khaleeji, ISO 639-3 afb). We compared each dialect sentence with the MSA sentence on the same line: 1,339 of 2,009 Najdi sentences are identical to MSA, against 2 arz_Arab, 1 apc_Arab, 41 acm_Arab. So the Najdi set is mostly MSA. Gulf Arabic keeps the 49 Tatoeba sentences tagged afb, and Arabizi keeps 39 Latin-script Tatoeba sentences from the Gulf, Egyptian and Levantine sets. 1 Tatoeba sentence was left out after review (reason in the JSON).
  • Other texts: 4 from Tatoeba (CC BY 2.0 FR), 30 from Wikipedia (CC BY-SA 4.0), 3 from Project Gutenberg (Public domain (USA)), 4 from Wikidata (CC0 1.0). Wikipedia gives the lead section of the same three articles (Water, Moon, Internet) in eight language editions, plus code blocks from programming articles. Gutenberg gives the opening of three novels. Wikidata gives entity JSON, pretty-printed and minified. English tokens per 1,000 words, the site's headline figure, comes from the English Wikipedia and Gutenberg prose only.
  • Uncertainty: for each tokenizer and parallel set, 10,000 bootstrap resamples of the sentence pairs (with replacement, the same pairs on both sides), seeded so a rerun gives the same numbers (seed 20261115); the interval is the 2.5th to 97.5th percentile of the pooled ratio. It covers sampling of sentences, not the choice of corpus: news-style FLORES-200 sentences and short everyday Tatoeba sentences give different ratios (see the change note below).
  • Tokenizers: the official tokenizer file at the commits listed above; GPT uses OpenAI's o200k_base. Gemma 4 uses Google's base-model tokenizer, since BroadNet has not published one for its build. Counts exclude chat-template and special tokens, so a real request adds a few tokens per message.
  • Gated files: Meta (Llama 3.3, Llama 4) and Google (Gemma 3) gate their repos. The script downloads the same file from an ungated mirror, computes its SHA-256 and size, rebuilds the Git LFS pointer, hashes it as a Git blob, and stops unless that ID equals the one the official repo lists for tokenizer.json at the pinned commit. The Hub API lists that ID without an account, so anyone can repeat the check.
  • Kimi: Moonshot publishes no tokenizer.json; the official tokenizer is tiktoken.model plus the split regex in tokenization_kimi.py. The script checks that regex is unchanged and counts with a tiktoken-compatible encoder.
  • Checks: in version 3, every count for the 11 added tokenizers matched an independent reference on all texts: the Rust tokenizers library for the tokenizer.json files and Python tiktoken for Kimi. For version 4 the FLORES-200 counts were recomputed independently with the same Python libraries for every tokenizer and language, sentence by sentence, and matched; the texts kept from version 3 gave the same counts, words and characters as before.
  • Ratio against English: for parallel sets, tokens ÷ tokens of the English side. The CSV also gives every text's tokens per word ÷ English prose tokens per word (same tokenizer).
  • Update frequency: token counts are re-measured when a tokenizer or the corpus changes; costs are recomputed on every build from BroadNet's pricing.json.
  • Limits: FLORES-200 is translated text (mostly from English, so the English side is the original), in a formal news and encyclopedia register; dialect translations follow MSA closely in places, and Najdi almost entirely. Chat, support tickets and code-mixed text will differ. The Gulf Arabic and Arabizi sets are small. Treat ratios as indicative and test your own text.

Licensing: what you may reuse

Our token, word and character counts, ratios, intervals, costs, tables and method are CC BY 4.0. FLORES-200 sentences are CC BY-SA 4.0 (NLLB Team et al., No Language Left Behind: Scaling Human-Centered Machine Translation (2022), arXiv:2207.04672; FLORES-200 evaluation benchmark.). Our CC BY 4.0 files contain no FLORES-200 text: only our own token, word and character counts for each language, with the corpus identified by its pinned tarball (SHA-256 b8b0b76783024b85797e5cc75064eb83fc5288b41e9654dabc7be6ae944011f6) and splits (dev, devtest). Get the sentences from https://dl.fbaipublicfiles.com/nllb/flores200_dataset.tar.gz. Cite FLORES-200 as: NLLB Team et al., No Language Left Behind: Scaling Human-Centered Machine Translation (2022), arXiv:2207.04672; FLORES-200 evaluation benchmark. The Tatoeba, Wikipedia, Gutenberg and Wikidata texts in the JSON keep their own licences, given per text.

What changed in version 4

Version 4 replaces the short Tatoeba sentence sets (33 to 73 sentences, 450 to 1,517 characters per language) with the 2,009 aligned FLORES-200 sentences for every language FLORES-200 covers, and adds Najdi and Mesopotamian Arabic. Gulf Arabic and Arabizi keep their Tatoeba sets: FLORES-200 has no equivalent. Each ratio now has a 95% interval. The tokenizers, the English prose texts and the code and JSON texts are unchanged, so English tokens per 1,000 words (1,290) did not move. Ratios moved because the sample is 28 to 61 times as many sentences and in a different register, not because a version 3 count was wrong: version 3 measured what its sentences contained. Short everyday sentences gave lower Arabic and Hindi ratios than news-style FLORES-200 text. Old against new, five core tokenizers:

SetSentences v3 → v4DeepSeek V4.1 FlashGLM 5.3Qwen3.8 27BGemma 4 VL 26BGPT (o200k)
Arabic MSA Tatoeba → FLORES-20051 → 2,0091.41× → 1.62× (1.61–1.63)1.53× → 1.77× (1.76–1.78)1.16× → 1.31× (1.30–1.32)1.22× → 1.47× (1.46–1.48)1.29× → 1.38× (1.37–1.39)
Gulf Arabic Tatoeba → Tatoeba49 → 491.44× → 1.44× (1.34–1.53)1.57× → 1.57× (1.47–1.68)1.21× → 1.21× (1.13–1.29)1.22× → 1.22× (1.12–1.32)1.32× → 1.32× (1.22–1.42)
Egyptian Arabic Tatoeba → FLORES-20038 → 2,0091.42× → 1.63× (1.61–1.64)1.53× → 1.75× (1.74–1.77)1.24× → 1.35× (1.34–1.36)1.21× → 1.44× (1.42–1.45)1.32× → 1.41× (1.40–1.42)
Levantine Arabic Tatoeba → FLORES-20033 → 2,0091.41× → 1.58× (1.57–1.59)1.50× → 1.70× (1.68–1.71)1.16× → 1.29× (1.28–1.30)1.09× → 1.41× (1.40–1.43)1.26× → 1.35× (1.34–1.36)
Najdi Arabic new → FLORES-200— → 2,009— → 1.63× (1.61–1.64)— → 1.77× (1.76–1.79)— → 1.31× (1.30–1.32)— → 1.47× (1.46–1.49)— → 1.39× (1.38–1.40)
Mesopotamian Arabic new → FLORES-200— → 2,009— → 1.62× (1.61–1.63)— → 1.75× (1.74–1.77)— → 1.32× (1.31–1.33)— → 1.46× (1.45–1.47)— → 1.39× (1.38–1.40)
Arabizi Tatoeba → Tatoeba39 → 391.50× → 1.50× (1.40–1.61)1.48× → 1.48× (1.38–1.59)1.38× → 1.38× (1.29–1.47)1.30× → 1.30× (1.21–1.39)1.45× → 1.45× (1.35–1.55)
French Tatoeba → FLORES-20046 → 2,0091.35× → 1.54× (1.53–1.55)1.28× → 1.50× (1.49–1.51)1.20× → 1.38× (1.37–1.39)1.19× → 1.40× (1.39–1.41)1.24× → 1.36× (1.35–1.37)
Spanish Tatoeba → FLORES-20046 → 2,0091.18× → 1.50× (1.48–1.51)1.10× → 1.44× (1.43–1.45)1.05× → 1.32× (1.31–1.33)0.96× → 1.29× (1.29–1.30)1.09× → 1.32× (1.31–1.33)
Hindi Tatoeba → FLORES-20073 → 2,0092.49× → 2.81× (2.79–2.83)3.67× → 4.33× (4.29–4.37)1.74× → 2.02× (2.00–2.03)1.21× → 1.31× (1.31–1.32)1.59× → 1.58× (1.56–1.59)
Urdu Tatoeba → FLORES-20046 → 2,0092.43× → 2.37× (2.36–2.39)2.82× → 2.75× (2.73–2.78)1.82× → 1.98× (1.97–2.00)1.24× → 1.53× (1.52–1.54)1.65× → 1.66× (1.65–1.67)
Chinese Tatoeba → FLORES-20037 → 2,0090.78× → 0.95× (0.94–0.96)0.81× → 0.96× (0.95–0.97)0.79× → 0.92× (0.91–0.93)0.89× → 1.09× (1.08–1.10)1.09× → 1.26× (1.25–1.27)

All 16 tokenizers, old and new, are in the JSON (changes). Version 3 (measured 2026-11-15) is superseded by this page.

What changed in version 3

Version 3 adds 11 tokenizers used by widely hosted open-weight models: gpt-oss (o200k_harmony), Llama 3.3 70B, Llama 4 Maverick, Kimi K3, MiniMax M3, Gemma 3 27B, Phi-4, Mistral Small 3.1, Ministral 3 14B, Nemotron 3 Ultra, Qwen3 32B. The corpus and the five v2 tokenizers were unchanged, and so were the headline averages, which use the five core tokenizers. Version 2 replaced the 12 short texts of version 1 with 52 openly licensed texts; English prose went from 1,142 to 1,290 tokens per 1,000 words.

What do the CSV columns mean?

ColumnMeaning
text_idText ID; the JSON "corpus" holds the full text, source, authors and sentence or revision IDs.
textShort label for the text (the text itself is only in the JSON).
groupLanguage or text kind, e.g. ar-msa, arabizi, code-python, json.
kindparallel (FLORES-200 or Tatoeba sentences), parallel-en (their English side), prose, code or json.
sourceWhere the text comes from: FLORES-200, Tatoeba, Wikipedia, Project Gutenberg or Wikidata.
licenseLicence of the source text (it is not relicensed). FLORES-200 text is not in our files at all; only our counts are.
urlSource link: the pinned FLORES-200 page, first Tatoeba sentence ID, or the exact page revision.
sentencesParallel texts: number of sentences (FLORES-200: 997 dev + 1,012 devtest).
charsUnicode code points.
wordsWord-like segments from Intl.Segmenter.
tokenizerTokenizer ID: deepseek, glm, qwen, gemma, gpt (o200k_base), gptoss (o200k_harmony), llama3, llama4, kimi, minimax, gemma3, phi4, mistral, ministral, nemotron or qwen3. The JSON "tokenizers" list gives each repo and pinned commit.
coreyes for the five v2 tokenizers behind the site's headline averages and token tools; no for the 11 added in v3.
tokensToken count, no special or chat-template tokens.
tokens_per_wordtokens ÷ words.
tokens_per_chartokens ÷ chars.
vs_english_per_wordtokens_per_word ÷ English prose tokens per word, same tokenizer.
vs_english_parallelParallel texts only: tokens ÷ tokens of the English translation of the same sentences.
vs_english_parallel_ci95_lowLower end of the 95% bootstrap interval for vs_english_parallel (10,000 seeded resamples of sentences).
vs_english_parallel_ci95_highUpper end of that interval.
broadnet_modelBroadNet model that uses this tokenizer, if any.
input_usd_per_1m_charstokens_per_char × that model's input price per 1M tokens (BroadNet pricing.json at build time).

How to cite this dataset

Token counts, ratios, cost figures, tables and method: CC BY 4.0. Free to use, share and adapt with attribution and a link to this page. FLORES-200 text is not included and stays CC BY-SA 4.0; when you use the FLORES-200 ratios, also cite NLLB Team et al., No Language Left Behind: Scaling Human-Centered Machine Translation (2022), arXiv:2207.04672; FLORES-200 evaluation benchmark.

The source texts in the JSON download are not relicensed: each keeps its source licence, given per text. Wikipedia texts are CC BY-SA 4.0, so a copy or adaptation of those texts must stay under CC BY-SA 4.0 and credit Wikipedia contributors (the revision link lists them). Tatoeba sentences are CC BY 2.0 FR: credit the authors listed per text (each sentence ID links to its author). Wikidata JSON is CC0; the Project Gutenberg excerpts are public domain. FLORES-200 sentences (CC BY-SA 4.0, NLLB Team, Meta) are not in the download at all: only our counts are, with the corpus pinned and linked.

Plain text

BroadRouter (2026). Tokenizer efficiency across languages: 16 tokenizers from DeepSeek, GLM, Qwen, Gemma, OpenAI, Meta, Moonshot, MiniMax, Microsoft, Mistral and NVIDIA (version 4.0) [Data set]. BroadNet Technologies. https://broadrouter.com/data/tokenizer-efficiency/

BibTeX

@misc{broadrouter_tokenizer_efficiency,
  author       = {BroadRouter},
  title        = {Tokenizer efficiency across languages: 16 tokenizers from DeepSeek, GLM, Qwen, Gemma, OpenAI, Meta, Moonshot, MiniMax, Microsoft, Mistral and NVIDIA},
  year         = {2026},
  version      = {4.0},
  publisher    = {BroadNet Technologies},
  howpublished = {\url{https://broadrouter.com/data/tokenizer-efficiency/}},
  note         = {Data set, data as of 2026-11-15},
}

Questions

How many tokens is 1,000 words?

About 1,290 tokens for English prose, averaged over the five core tokenizers (1,245 to 1,320 across all 16). Arabic, Hindi, code and JSON need more. Convert any amount with the tokens to words converter.

Which tokenizer is most efficient for Arabic?

For Modern Standard Arabic, Ministral 3 14B needed the fewest tokens relative to the same 2,009 English sentences (1.25×, 95% interval 1.24–1.25) and Phi-4 the most (3.03×, 3.01–3.05). Fewer tokens is not the same as cheaper: see what the same text costs by provider, or measure your own text with the Arabic token counter.

Which tokenizer is most efficient for Hindi and Urdu?

Gemma 4 VL 26B needed the fewest tokens for Hindi (1.31× the English count) and Phi-4 the most (4.78×). For Urdu the range is 1.53× (Gemma 4 VL 26B) to 4.36× (Phi-4).

Does Arabizi use more tokens than Arabic script?

Arabizi costs 1.30× (Gemma 4 VL 26B) to 1.55× (Kimi K3) the English token count. Latin letters reuse English word pieces, but the digits used for Arabic sounds (2, 3, 7) and non-standard spelling split words further.

Why does Arabic cost more tokens than English?

Tokenizers learn their vocabulary from training text. When a vocabulary holds fewer Arabic word pieces, Arabic words split into more tokens, and API bills are per token. Dialect spelling and Arabizi push the count higher.

How sure are these ratios?

Each ratio against English comes with a 95% interval from a bootstrap over sentences (10,000 resamples, fixed seed). On the 2,009 FLORES-200 sentences per language no interval is wider than 0.08 (ratio points). The Gulf Arabic and Arabizi sets have under 50 Tatoeba sentences each, and their intervals are up to 0.34 wide, so treat those two as rough.

Can I reproduce these numbers?

Yes. Download FLORES-200 from Meta (the SHA-256 is on this page), then in the site repository run node scripts/dataset-tokenizer-flores.mjs and node scripts/dataset-tokenizer-efficiency.mjs with FLORES_DIR set. The JSON download includes every other text with its source, licence and revision or sentence IDs, and every tokenizer with its repo and pinned commit. The script re-checks the gated Meta and Google tokenizer files against the official repos before counting.

Can I use this data?

Yes. The token counts, ratios, intervals and costs are CC BY 4.0; cite them as shown under "How to cite". FLORES-200 sentences are CC BY-SA 4.0 and are not in our files; we publish only our counts, with the corpus pinned and linked. The texts in the JSON keep their own licences: Wikipedia CC BY-SA 4.0 (share-alike), Tatoeba CC BY 2.0 FR (credit the authors listed per text), Wikidata CC0 and the Gutenberg excerpts public domain.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity