Byte pair encoding is the recipe behind most LLM tokenizers: learn which pieces of text occur together often, and give each frequent piece its own token.
How it works
Sennrich, Haddow and Birch (2015) applied BPE, originally a compression method, to translation. Their point was that names, compounds and rare words can be built from smaller units, so a model does not need a dictionary entry for every word.
Training a BPE vocabulary follows a simple loop, shown step by step in Hugging Face’s documentation:
- Count word frequencies in a training corpus, for example
hug10,pug5,pun12,bun4,hugs5. - Start with single characters:
b g h n p s u. - Find the most frequent adjacent pair.
u+gappears 20 times, so addug. - Repeat.
u+n(16 times) becomesun. - Stop at the target vocabulary size, which equals the base symbols plus the number of merges.
To encode new text, the tokenizer applies the learned merges in the same order. Encoding is deterministic: the same text always gives the same tokens.
Byte-level BPE starts from the 256 byte values instead of characters. Every possible string is then encodable. GPT-2’s vocabulary was 256 bytes + 50,000 merges + 1 end-of-text token = 50,257.
Why it matters
BPE explains several things you see on an API bill:
- Common English words are one token, because their pairs were merged early.
- Rare words, typos and non-Latin scripts split into more tokens. In UTF-8 an Arabic letter takes 2 bytes, so a byte-level tokenizer that learned few Arabic merges spends more tokens per word.
- Counts differ by model. Each model learns its own merges from its own data.
Our measurement on 2026-11-14 shows the effect. The same Gulf-dialect chat of 117 words was 276 tokens on GLM 5.3 and 229 on Qwen3.8 27B. That is 276 / 117 x 100 = 235.9 versus 229 / 117 x 100 = 195.7 tokens per 100 words. The English translation of that chat cost 202 and 208 tokens, nearly the same.
On BroadNet Inference
You do not choose a tokenizer on BroadNet; each model uses its own. Compare counts for your real text in the Arabic token counter before you pick a model for Arabic traffic, and read tokens per word in Arabic and English for the full table. Remember that a token count, not a word count, sets the price.