Models do not see letters or words. They see a sequence of integers, each standing for a piece of text from a fixed vocabulary. Those pieces are tokens.
How it works
A tokenizer splits text before the model reads it. Most modern models use subword methods such as byte pair encoding (BPE). Common words stay whole; rare words break into parts. Hugging Face’s example: annoyingly may become annoying and ly.
Byte-level BPE starts from 256 byte values, so any input can be encoded, including emoji and every script.
For English, OpenAI’s rule of thumb is about 4 characters or 0.75 words per token, so 100 tokens is roughly 75 words. Our own measurement of English prose on five current tokenizers came to about 129 tokens per 100 words, or 1,290 tokens per 1,000 words, close to the rule of thumb. Either ratio is only an estimate:
- other languages, including Arabic, often split into more tokens per word on tokenizers trained mostly on English;
- code, numbers and whitespace tokenize differently from prose;
- every model family has its own vocabulary.
To estimate before you send, paste text into the token counter. The API reports the real count in the usage block of each response:
"usage": { "prompt_tokens": 1200, "completion_tokens": 340, "total_tokens": 1540 }
Why it matters
Tokens are the unit behind almost every number on an LLM API:
- Price. Per-token APIs bill per 1M input and output tokens, often at different rates.
- Limits. The context window and
max_tokensare measured in tokens, and many rate limits count tokens per minute. - Speed. Generation speed is quoted in tokens per second.
The same text can be a different number of tokens on different models, especially in Arabic. The tokenizer efficiency dataset measures tokens per 100 words for each model in English, Arabic and code.
On BroadNet Inference
Chat models are priced per 1M tokens. For DeepSeek V4.1 Flash, new input costs $0.175, cached input costs $0.0055 and output costs $0.52. BroadNet Gemma 4 VL 26B costs $0.10 input and $0.22 output. Every model’s rates are on the pricing page.
On a flat plan, tokens are not metered at all: a session of DeepSeek V4.1 Flash has no token meter, and the practical limit is speed, about 100 tokens per second. Speech-to-text models bill per audio minute, not per token.
Common mistakes
- Estimating cost from word counts for non-English text. Measure with
usageon real samples. - Forgetting that reasoning models also spend output tokens on thinking. See reasoning tokens.