---
title: "Tokens per word: Arabic vs English, and how to measure"
description: "Why Arabic often uses more LLM tokens than English for the same meaning, why tokens per word misleads, and a script to measure your own ratio from the usage field."
url: https://broadrouter.com/blog/tokens-per-word-arabic-english/
lang: en
updated: 2026-11-14
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Tokens per word in Arabic vs English: why Arabic costs more

By the BroadNet Inference team · 6 min read · Published 2026-11-14 · Updated 2026-11-16

There is no single figure: it depends on each model's tokenizer. In BroadRouter's measured dataset, the same Modern Standard Arabic text needs 1.31× (Qwen3.8 27B) to 1.77× (GLM 5.3) the tokens of its English translation, and 1,000 words of English prose come to about 1,290 tokens on average. Measure your own ratio from usage.prompt\_tokens.

## Key takeaways

-   Tokens per word is a property of each model's tokenizer, not of the language alone.
-   Research on many tokenizers finds the same content often costs more tokens in Arabic than in English.
-   Compare parallel text (the same meaning in both languages), not tokens per word: one Arabic word can carry several English words.
-   Measure with usage.prompt\_tokens on the exact model you will use, subtracting a baseline for message overhead.
-   Diacritics, tatweel and mixed scripts change counts; normalise input consistently.

**On this page**

1.  How many more tokens does Arabic need? Measured numbers
2.  What does a tokenizer do?
3.  Why does Arabic often use more tokens?
4.  Why is tokens per word the wrong metric?
5.  How do you measure your own ratio?
6.  What does it mean for cost and context?
7.  How can you reduce Arabic token counts?

If you build an Arabic product on an LLM API, your bill is counted in tokens, not words. And the same message, written in Arabic, often turns into more tokens than its English translation. That affects cost, context limits and latency. This post explains why it happens, why “tokens per word” is a misleading number, and how to measure the real ratio for your own content on the model you use.

## How many more tokens does Arabic need? Measured numbers

We measured this with each model’s real tokenizer on parallel texts (the same meaning in English and Arabic). The full table, 95% intervals and method are in the free [tokenizer efficiency dataset](https://broadrouter.com/data/tokenizer-efficiency/) (CSV and JSON, CC BY 4.0, version 4, measured 2026-11-15):

| Tokenizer | Arabic ÷ English, Modern Standard (2,009 FLORES-200 sentences) | Arabic ÷ English, Gulf dialect (49 Tatoeba sentences) |
| --- | --- | --- |
| Qwen3.8 27B | 1.31× | 1.21× |
| GPT (o200k) | 1.38× | 1.32× |
| Gemma 4 VL 26B | 1.47× | 1.22× |
| DeepSeek V4.1 Flash | 1.62× | 1.44× |
| GLM 5.3 | 1.77× | 1.57× |

English prose (Wikipedia and Project Gutenberg) came to about 127 to 132 tokens per 100 words across the five. The Modern Standard Arabic ratios come from the same 2,009 FLORES-200 sentences in both languages (formal news, travel and encyclopedia text from English-language Wikimedia articles, professionally translated). FLORES-200 has no Gulf Arabic, so the Gulf column uses 49 short Tatoeba sentence pairs and is much less precise; the dataset page gives an interval for every figure. None of this is your traffic, so use the ratios for planning and measure your own content before you budget. The [Arabic token counter](https://broadrouter.com/tools/arabic-token-counter/) counts a pasted text on each tokenizer, and [tokens to words](https://broadrouter.com/tools/tokens-to-words/) converts in both directions.

To turn these ratios into money, our [real cost by language dataset](https://broadrouter.com/data/real-cost-by-language/) prices 1M characters of each language on every model we could match to a measured tokenizer, across providers. A cheaper price per token does not always mean a cheaper Arabic bill.

## What does a tokenizer do?

Before a model sees text, a tokenizer splits it into tokens drawn from a fixed vocabulary. Most modern tokenizers use subword methods such as byte-pair encoding (BPE), WordPiece or SentencePiece, described in Hugging Face’s [summary of the tokenizers](https://huggingface.co/docs/transformers/tokenizer_summary). Frequent strings in the training data become single tokens; rare strings are broken into smaller pieces, down to single bytes if necessary.

The vocabulary is learned from a training corpus. Whatever languages dominate that corpus get the most efficient encoding. Each model family has its own tokenizer, so DeepSeek, Qwen, GLM and Gemma models can split the same Arabic sentence into quite different numbers of tokens.

## Why does Arabic often use more tokens?

### 1\. Training data imbalance

Many tokenizers were trained on text where English is heavily over-represented. Common English words become one token each; many common Arabic words are split into several pieces. Petrov et al. (2023) measured this across many tokenizers and languages and found large differences in how many tokens the same content needs, with English typically among the cheapest. Ahia et al. (2023) reached a similar conclusion for commercial APIs and noted the direct effect on what speakers of different languages pay.

### 2\. Arabic morphology

Arabic attaches conjunctions, prepositions, the article and pronouns directly to words. One written word can carry what English spreads over several:

| Arabic | Rough English |
| --- | --- |
| وسيكتبونها | and they will write it |
| بالمدرسة | in the school |
| فقلنا | so we said |

A tokenizer that has not seen a particular combination often enough splits it into pieces. That raises tokens per **word**, even when tokens per **meaning** are reasonable.

### 3\. Script details

Diacritics (tashkeel), tatweel (the elongation character), different forms of alef and hamza, and Arabic-Indic digits are all extra characters or variants the tokenizer must encode. Mixed Arabic and Latin text, and Arabizi (Arabic written in Latin letters with digits), can split in unexpected ways too. See [how LLMs handle Arabizi](https://broadrouter.com/blog/arabizi-llm-handling/).

## Why is tokens per word the wrong metric?

Because one Arabic word often equals several English words, “Arabic uses N tokens per word” and “English uses M tokens per word” are not comparable. A fairer question is: **how many tokens does the same content need in each language?** Use parallel text, the same message written in both languages, and compare total tokens. That ratio is what changes your bill.

OpenAI’s often-quoted rule of thumb, about four characters or three quarters of a word per token, is given for English on OpenAI’s own tokenizers. It does not carry over to Arabic or to other model families, so do not use it for Arabic budgets.

## How do you measure your own ratio?

The usage field is the ground truth: it is what you are billed on. This script sends parallel texts to a model and reads `usage.prompt_tokens`. It subtracts a baseline so the fixed chat-template overhead does not distort short texts. It uses [Gemma 4 VL 26B](https://broadrouter.com/models/gemma-4-vl-26b/), which works on the free starting credit and does not reason; swap in the model you will actually use.

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
MODEL = "broadnet/gemma-4-26b-a4b"

def prompt_tokens(text):
    r = client.chat.completions.create(
        model=MODEL, max_tokens=1,
        messages=[{"role": "user", "content": text}],
    )
    return r.usage.prompt_tokens

baseline = prompt_tokens(".")
pairs = [
    ("Your order has shipped and will arrive on Thursday.",
     "تم شحن طلبك وسيصل يوم الخميس."),
    ("Please send a photo of the invoice so we can check the total.",
     "يرجى إرسال صورة الفاتورة حتى نتحقق من المبلغ الإجمالي."),
]

en_total = ar_total = 0
for en, ar in pairs:
    en_t = prompt_tokens(en) - baseline
    ar_t = prompt_tokens(ar) - baseline
    en_total += en_t; ar_total += ar_t
    print(f"EN {en_t:4d} | AR {ar_t:4d} | ratio {ar_t / max(1, en_t):.2f}")

print(f"Overall Arabic/English token ratio: {ar_total / max(1, en_total):.2f}")
```

To get a number you can trust:

1.  **Use 50–200 real pairs** from your own domain: support messages, product descriptions, invoices. Short demo sentences are noisy.
2.  **Use the dialect your users write.** Gulf, Egyptian and Levantine text can tokenize differently from Modern Standard Arabic.
3.  **Run it per model.** A ratio measured on one model says little about another. If you are choosing between models, run the same pairs on each; [choosing an LLM for Arabic](https://broadrouter.com/blog/choosing-an-llm-for-arabic/) covers quality, which matters as much as token count.
4.  **Measure output too.** Ask each model to answer in Arabic and in English and compare `completion_tokens`. Output costs more per token than input on every model, so the output ratio can matter more.

If the model’s tokenizer is published (for example on Hugging Face), you can also count tokens offline with that tokenizer. The API’s usage field is still the final word, because it includes the chat template the server applies.

### Reading the results

When you have a ratio for each model, three patterns are common:

-   **Close to 1.** The tokenizer handles your Arabic well. Arabic and English traffic cost about the same for the same content.
-   **Clearly above 1.** Arabic costs more per message. Budget for it in forecasts and in context limits, and consider whether another model handles your Arabic more efficiently at similar quality.
-   **Very different across your samples.** Usually a sign of mixed content: diacritised text, Arabizi, numbers, URLs or code inside Arabic messages. Split your samples by type and measure each separately.

Keep the script and rerun it when you change models or when a provider updates a model version, because a new tokenizer changes every count.

## What does it mean for cost and context?

Cost scales with tokens, so a higher Arabic ratio raises the bill by the same factor for the Arabic part of your traffic. Using BroadNet’s prices, Arabic input on [Qwen3.8 27B](https://broadrouter.com/models/qwen3.8-27b/) costs $0.175 per 1M tokens and output $2.17; on Gemma 4 VL 26B, $0.10 and $0.22. The price per token is the same in any language; only the token count changes. Feed your measured ratio into the [monthly cost method](https://broadrouter.com/blog/estimate-llm-api-cost-before-launch/).

Context limits shrink the same way. Gemma 4 VL 26B accepts 32,768 tokens per request; a long Arabic document fills that sooner than its English translation would. DeepSeek V4.1 Flash and GLM 5.3 accept 1,048,576.

## How can you reduce Arabic token counts?

-   **Normalise consistently.** Remove tatweel, and strip diacritics if your task does not need them. Apply the same normalisation to every request so caching still works.
-   **Keep instructions short.** A long system prompt is resent on every turn; tight wording saves tokens in both languages.
-   **Cache the stable prefix.** On models with a cached rate, repeated system prompts and documents cost far less. [How caching works](https://broadrouter.com/blog/cached-input-tokens-explained/).
-   **Ask for concise answers.** “Answer in two sentences” saves more than any tokenizer trick.

For Arabic speech rather than text, transcription is priced per audio minute, not per token, so tokenization does not affect that part of the bill; see the [Arabic speech-to-text page](https://broadrouter.com/blog/arabic-speech-to-text-api-buyers-guide/).

Free tools for this

-   [LLM API cost calculator](https://broadrouter.com/tools/llm-api-cost-calculator/) Price a workload on every model at once, with cache hits, reasoning and a flat-plan check.
-   [Flat-rate vs per-token LLM cost calculator](https://broadrouter.com/tools/flat-rate-vs-per-token-calculator/) Break-even between an unlimited plan and paying per token, per agent.
-   [Prompt caching savings calculator](https://broadrouter.com/tools/prompt-caching-calculator/) Bill with and without caching, by prefix share and hit rate.
-   [Prompt token diff](https://broadrouter.com/tools/prompt-token-diff/) Two prompt versions side by side: changed lines, exact token delta and the monthly cost difference.

[All free tools](https://broadrouter.com/tools/)

## Terms in this article

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[Tokenizer](https://broadrouter.com/glossary/tokenizer/)

A tokenizer is the component that converts text into the sequence of integer token IDs a language model reads, and converts generated IDs back into text. Each model ships its own tokenizer, so the same text can produce a different number of tokens on different models.

[Modern Standard Arabic (MSA)](https://broadrouter.com/glossary/msa/)

Modern Standard Arabic (MSA) is the standardized form of Arabic used in writing, news, education and formal speech across the Arab world. It is nobody's native everyday language: people speak regional dialects such as Egyptian or Gulf Arabic and switch to MSA in formal settings.

[Arabic diacritics (tashkeel)](https://broadrouter.com/glossary/arabic-diacritics/)

Arabic diacritics, or tashkeel, are small marks written above or below letters to show short vowels, doubled consonants and the absence of a vowel. They are optional: most Arabic text, including news, omits them, so readers and software infer the vowels from context.

[Arabizi](https://broadrouter.com/glossary/arabizi/)

Arabizi is Arabic written informally in Latin letters, using digits for Arabic sounds with no Latin equivalent, such as 3 for ع. It is common in chat and social media, is mostly dialectal, has no standard spelling, and is often mixed with English or French.

[Context window](https://broadrouter.com/glossary/context-window/)

A context window is the maximum number of tokens a language model can work with in a single request. It covers the prompt, conversation history, tool definitions, images and the generated output, including any reasoning tokens. Anything that does not fit must be cut, summarised or split across several requests.

[Arabic text normalization](https://broadrouter.com/glossary/arabic-text-normalization/)

Arabic text normalization is the step that maps spelling variants of the same word to one form before matching: removing diacritics and tatweel, unifying alef forms, and often mapping ى to ي and ة to ه. It is applied to search keys and evaluation, not to text you display.

[System prompt](https://broadrouter.com/glossary/system-prompt/)

A system prompt is the instruction text an application sends to a language model ahead of the conversation, usually as a message with the system or developer role. It sets the model's role, rules, tone and output format, and the model is meant to follow it over conflicting user messages.

## Questions

### How many tokens is one Arabic word?

It depends on the model. In BroadRouter's tokenizer dataset, Modern Standard Arabic took 183.0 (Qwen3.8 27B) to 243.9 (GLM 5.3) tokens per 100 words across five tokenizers. Compare parallel texts rather than single words, and measure on the model you use.

### How many tokens is 1,000 English words?

About 1,269 to 1,320 tokens of plain English prose across the five tokenizers in BroadRouter's measured dataset (1,290 on average). Technical text, code and other languages need more, so count your own text.

### Does removing diacritics reduce tokens?

Usually, because diacritics are extra characters the tokenizer must encode. Only remove them if your task does not need them; for Quran, poetry or pronunciation they carry meaning.

### Is the 4-characters-per-token rule valid for Arabic?

No. OpenAI gives that rule of thumb for English text on its own tokenizers. It does not transfer to Arabic or to other models' tokenizers.

### Do Arabic tokens cost more per token?

No. A token costs the same whatever language it encodes. Arabic costs more only when the same content splits into more tokens.

## Keep reading

-   [Choosing an LLM API for Arabic: what to test](https://broadrouter.com/blog/choosing-an-llm-for-arabic/)
-   [LLM API pricing explained: input, cached, output and reasoning](https://broadrouter.com/blog/llm-api-pricing-explained/)
-   [Arabic prompt engineering: register, RTL, numerals and mixed text](https://broadrouter.com/blog/arabic-prompt-engineering/)
-   [Arabizi and LLMs: handling Franco-Arabic with numbers like 3 and 7](https://broadrouter.com/blog/arabizi-llm-handling/)
-   [Add diacritics to Arabic text with an LLM: prompt, checks, DER](https://broadrouter.com/blog/arabic-diacritization-llm/)
-   [LLM API price changes in October 2026: 3 rises, no cuts](https://broadrouter.com/blog/llm-api-price-changes-october-2026/)
-   [Do LLM API prices only go down? 52 rises and how to budget for them](https://broadrouter.com/blog/llm-api-price-increases/)
-   [Guide: Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-tokens-per-word-arabic-english&utm_content=page-cta) ·

**Sources**

-   [Petrov et al., Language Model Tokenizers Introduce Unfairness Between Languages (arXiv 2305.15425, checked 2026-11-16)](https://arxiv.org/abs/2305.15425)
-   [Ahia et al., Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models (arXiv 2305.13707, checked 2026-11-16)](https://arxiv.org/abs/2305.13707)
-   [Hugging Face Transformers, tokenization algorithms (checked 2026-11-16)](https://huggingface.co/docs/transformers/tokenizer_summary)
-   [OpenAI Help Center, what are tokens and how to count them](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them)
-   [BroadRouter tokenizer efficiency dataset, version 4 (measured 2026-11-15)](https://broadrouter.com/data/tokenizer-efficiency/)
-   [FLORES-200 evaluation benchmark (Meta, CC BY-SA 4.0), the parallel sentences behind the MSA ratios (pinned commit, checked 2026-11-16)](https://github.com/facebookresearch/flores/tree/a6c830c6e1051fb4ac1a44b32358f00463f332bd/flores200)
-   [BroadNet Inference API reference, usage fields (checked 2026-11-16)](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-tokens-per-word-arabic-english&utm_content=sources)
