BroadRouter
Get started

Byte pair encoding (BPE)

Updated

Byte pair encoding (BPE) is a subword tokenization algorithm that starts from single characters or bytes and repeatedly merges the most frequent adjacent pair into a new vocabulary entry, until it reaches a target vocabulary size. Most current LLM tokenizers use BPE or a byte-level variant.

Also called: byte pair encoding, byte-pair encoding, byte-level BPE, BPE tokenization

Byte pair encoding is the recipe behind most LLM tokenizers: learn which pieces of text occur together often, and give each frequent piece its own token.

How it works

Sennrich, Haddow and Birch (2015) applied BPE, originally a compression method, to translation. Their point was that names, compounds and rare words can be built from smaller units, so a model does not need a dictionary entry for every word.

Training a BPE vocabulary follows a simple loop, shown step by step in Hugging Face’s documentation:

  1. Count word frequencies in a training corpus, for example hug 10, pug 5, pun 12, bun 4, hugs 5.
  2. Start with single characters: b g h n p s u.
  3. Find the most frequent adjacent pair. u + g appears 20 times, so add ug.
  4. Repeat. u + n (16 times) becomes un.
  5. Stop at the target vocabulary size, which equals the base symbols plus the number of merges.

To encode new text, the tokenizer applies the learned merges in the same order. Encoding is deterministic: the same text always gives the same tokens.

Byte-level BPE starts from the 256 byte values instead of characters. Every possible string is then encodable. GPT-2’s vocabulary was 256 bytes + 50,000 merges + 1 end-of-text token = 50,257.

Why it matters

BPE explains several things you see on an API bill:

  • Common English words are one token, because their pairs were merged early.
  • Rare words, typos and non-Latin scripts split into more tokens. In UTF-8 an Arabic letter takes 2 bytes, so a byte-level tokenizer that learned few Arabic merges spends more tokens per word.
  • Counts differ by model. Each model learns its own merges from its own data.

Our measurement on 2026-11-14 shows the effect. The same Gulf-dialect chat of 117 words was 276 tokens on GLM 5.3 and 229 on Qwen3.8 27B. That is 276 / 117 x 100 = 235.9 versus 229 / 117 x 100 = 195.7 tokens per 100 words. The English translation of that chat cost 202 and 208 tokens, nearly the same.

On BroadNet Inference

You do not choose a tokenizer on BroadNet; each model uses its own. Compare counts for your real text in the Arabic token counter before you pick a model for Arabic traffic, and read tokens per word in Arabic and English for the full table. Remember that a token count, not a word count, sets the price.

Questions

Who invented BPE for language models?

BPE began as a data compression algorithm. Sennrich, Haddow and Birch adapted it in 2015 to split rare words into subword units for neural machine translation, and LLM tokenizers grew from that work.

What is byte-level BPE?

A BPE tokenizer whose base vocabulary is the 256 possible byte values instead of characters, so any text, in any script, can be encoded without an unknown token. GPT-2 used it with 50,257 entries.

Why does BPE produce more tokens for Arabic than English?

Merges are learned from training data. If a tokenizer saw less Arabic, fewer Arabic pairs were merged, and Arabic words break into more pieces.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity