If you build an Arabic product on an LLM API, your bill is counted in tokens, not words. And the same message, written in Arabic, often turns into more tokens than its English translation. That affects cost, context limits and latency. This post explains why it happens, why “tokens per word” is a misleading number, and how to measure the real ratio for your own content on the model you use.
How many more tokens does Arabic need? Measured numbers
We measured this with each model’s real tokenizer on parallel texts (the same meaning in English and Arabic). The full table, 95% intervals and method are in the free tokenizer efficiency dataset (CSV and JSON, CC BY 4.0, version 4, measured 2026-11-15):
| Tokenizer | Arabic ÷ English, Modern Standard (2,009 FLORES-200 sentences) | Arabic ÷ English, Gulf dialect (49 Tatoeba sentences) |
|---|---|---|
| Qwen3.8 27B | 1.31× | 1.21× |
| GPT (o200k) | 1.38× | 1.32× |
| Gemma 4 VL 26B | 1.47× | 1.22× |
| DeepSeek V4.1 Flash | 1.62× | 1.44× |
| GLM 5.3 | 1.77× | 1.57× |
English prose (Wikipedia and Project Gutenberg) came to about 127 to 132 tokens per 100 words across the five. The Modern Standard Arabic ratios come from the same 2,009 FLORES-200 sentences in both languages (formal news, travel and encyclopedia text from English-language Wikimedia articles, professionally translated). FLORES-200 has no Gulf Arabic, so the Gulf column uses 49 short Tatoeba sentence pairs and is much less precise; the dataset page gives an interval for every figure. None of this is your traffic, so use the ratios for planning and measure your own content before you budget. The Arabic token counter counts a pasted text on each tokenizer, and tokens to words converts in both directions.
To turn these ratios into money, our real cost by language dataset prices 1M characters of each language on every model we could match to a measured tokenizer, across providers. A cheaper price per token does not always mean a cheaper Arabic bill.
What does a tokenizer do?
Before a model sees text, a tokenizer splits it into tokens drawn from a fixed vocabulary. Most modern tokenizers use subword methods such as byte-pair encoding (BPE), WordPiece or SentencePiece, described in Hugging Face’s summary of the tokenizers. Frequent strings in the training data become single tokens; rare strings are broken into smaller pieces, down to single bytes if necessary.
The vocabulary is learned from a training corpus. Whatever languages dominate that corpus get the most efficient encoding. Each model family has its own tokenizer, so DeepSeek, Qwen, GLM and Gemma models can split the same Arabic sentence into quite different numbers of tokens.
Why does Arabic often use more tokens?
1. Training data imbalance
Many tokenizers were trained on text where English is heavily over-represented. Common English words become one token each; many common Arabic words are split into several pieces. Petrov et al. (2023) measured this across many tokenizers and languages and found large differences in how many tokens the same content needs, with English typically among the cheapest. Ahia et al. (2023) reached a similar conclusion for commercial APIs and noted the direct effect on what speakers of different languages pay.
2. Arabic morphology
Arabic attaches conjunctions, prepositions, the article and pronouns directly to words. One written word can carry what English spreads over several:
| Arabic | Rough English |
|---|---|
| وسيكتبونها | and they will write it |
| بالمدرسة | in the school |
| فقلنا | so we said |
A tokenizer that has not seen a particular combination often enough splits it into pieces. That raises tokens per word, even when tokens per meaning are reasonable.
3. Script details
Diacritics (tashkeel), tatweel (the elongation character), different forms of alef and hamza, and Arabic-Indic digits are all extra characters or variants the tokenizer must encode. Mixed Arabic and Latin text, and Arabizi (Arabic written in Latin letters with digits), can split in unexpected ways too. See how LLMs handle Arabizi.
Why is tokens per word the wrong metric?
Because one Arabic word often equals several English words, “Arabic uses N tokens per word” and “English uses M tokens per word” are not comparable. A fairer question is: how many tokens does the same content need in each language? Use parallel text, the same message written in both languages, and compare total tokens. That ratio is what changes your bill.
OpenAI’s often-quoted rule of thumb, about four characters or three quarters of a word per token, is given for English on OpenAI’s own tokenizers. It does not carry over to Arabic or to other model families, so do not use it for Arabic budgets.
How do you measure your own ratio?
The usage field is the ground truth: it is what you are billed on. This script sends parallel texts to a model and reads usage.prompt_tokens. It subtracts a baseline so the fixed chat-template overhead does not distort short texts. It uses Gemma 4 VL 26B, which works on the free starting credit and does not reason; swap in the model you will actually use.
import os
from openai import OpenAI
client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
MODEL = "broadnet/gemma-4-26b-a4b"
def prompt_tokens(text):
r = client.chat.completions.create(
model=MODEL, max_tokens=1,
messages=[{"role": "user", "content": text}],
)
return r.usage.prompt_tokens
baseline = prompt_tokens(".")
pairs = [
("Your order has shipped and will arrive on Thursday.",
"تم شحن طلبك وسيصل يوم الخميس."),
("Please send a photo of the invoice so we can check the total.",
"يرجى إرسال صورة الفاتورة حتى نتحقق من المبلغ الإجمالي."),
]
en_total = ar_total = 0
for en, ar in pairs:
en_t = prompt_tokens(en) - baseline
ar_t = prompt_tokens(ar) - baseline
en_total += en_t; ar_total += ar_t
print(f"EN {en_t:4d} | AR {ar_t:4d} | ratio {ar_t / max(1, en_t):.2f}")
print(f"Overall Arabic/English token ratio: {ar_total / max(1, en_total):.2f}")
To get a number you can trust:
- Use 50–200 real pairs from your own domain: support messages, product descriptions, invoices. Short demo sentences are noisy.
- Use the dialect your users write. Gulf, Egyptian and Levantine text can tokenize differently from Modern Standard Arabic.
- Run it per model. A ratio measured on one model says little about another. If you are choosing between models, run the same pairs on each; choosing an LLM for Arabic covers quality, which matters as much as token count.
- Measure output too. Ask each model to answer in Arabic and in English and compare
completion_tokens. Output costs more per token than input on every model, so the output ratio can matter more.
If the model’s tokenizer is published (for example on Hugging Face), you can also count tokens offline with that tokenizer. The API’s usage field is still the final word, because it includes the chat template the server applies.
Reading the results
When you have a ratio for each model, three patterns are common:
- Close to 1. The tokenizer handles your Arabic well. Arabic and English traffic cost about the same for the same content.
- Clearly above 1. Arabic costs more per message. Budget for it in forecasts and in context limits, and consider whether another model handles your Arabic more efficiently at similar quality.
- Very different across your samples. Usually a sign of mixed content: diacritised text, Arabizi, numbers, URLs or code inside Arabic messages. Split your samples by type and measure each separately.
Keep the script and rerun it when you change models or when a provider updates a model version, because a new tokenizer changes every count.
What does it mean for cost and context?
Cost scales with tokens, so a higher Arabic ratio raises the bill by the same factor for the Arabic part of your traffic. Using BroadNet’s prices, Arabic input on Qwen3.8 27B costs $0.175 per 1M tokens and output $2.17; on Gemma 4 VL 26B, $0.10 and $0.22. The price per token is the same in any language; only the token count changes. Feed your measured ratio into the monthly cost method.
Context limits shrink the same way. Gemma 4 VL 26B accepts 32,768 tokens per request; a long Arabic document fills that sooner than its English translation would. DeepSeek V4.1 Flash and GLM 5.3 accept 1,048,576.
How can you reduce Arabic token counts?
- Normalise consistently. Remove tatweel, and strip diacritics if your task does not need them. Apply the same normalisation to every request so caching still works.
- Keep instructions short. A long system prompt is resent on every turn; tight wording saves tokens in both languages.
- Cache the stable prefix. On models with a cached rate, repeated system prompts and documents cost far less. How caching works.
- Ask for concise answers. “Answer in two sentences” saves more than any tokenizer trick.
For Arabic speech rather than text, transcription is priced per audio minute, not per token, so tokenization does not affect that part of the bill; see the Arabic speech-to-text page.
