---
title: "What is byte pair encoding (BPE) in LLM tokenizers? | BroadRouter"
description: "Byte pair encoding builds a tokenizer vocabulary by merging the most frequent symbol pairs. How BPE and byte-level BPE work, and why Arabic splits into more tokens."
url: https://broadrouter.com/glossary/bpe/
lang: en
updated: 2026-11-16
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Byte pair encoding (BPE)

Updated 2026-11-16

Byte pair encoding (BPE) is a subword tokenization algorithm that starts from single characters or bytes and repeatedly merges the most frequent adjacent pair into a new vocabulary entry, until it reaches a target vocabulary size. Most current LLM tokenizers use BPE or a byte-level variant.

Also called: byte pair encoding, byte-pair encoding, byte-level BPE, BPE tokenization

Byte pair encoding is the recipe behind most LLM [tokenizers](https://broadrouter.com/glossary/tokenizer/): learn which pieces of text occur together often, and give each frequent piece its own token.

## How it works

Sennrich, Haddow and Birch (2015) applied BPE, originally a compression method, to translation. Their point was that names, compounds and rare words can be built from smaller units, so a model does not need a dictionary entry for every word.

Training a BPE vocabulary follows a simple loop, shown step by step in Hugging Face’s documentation:

1.  Count word frequencies in a training corpus, for example `hug` 10, `pug` 5, `pun` 12, `bun` 4, `hugs` 5.
2.  Start with single characters: `b g h n p s u`.
3.  Find the most frequent adjacent pair. `u` + `g` appears 20 times, so add `ug`.
4.  Repeat. `u` + `n` (16 times) becomes `un`.
5.  Stop at the target [vocabulary size](https://broadrouter.com/glossary/vocabulary-size/), which equals the base symbols plus the number of merges.

To encode new text, the tokenizer applies the learned merges in the same order. Encoding is deterministic: the same text always gives the same tokens.

**Byte-level BPE** starts from the 256 byte values instead of characters. Every possible string is then encodable. GPT-2’s vocabulary was 256 bytes + 50,000 merges + 1 end-of-text token = 50,257.

## Why it matters

BPE explains several things you see on an API bill:

-   **Common English words are one token**, because their pairs were merged early.
-   **Rare words, typos and non-Latin scripts split into more tokens.** In UTF-8 an Arabic letter takes 2 bytes, so a byte-level tokenizer that learned few Arabic merges spends more tokens per word.
-   **Counts differ by model.** Each model learns its own merges from its own data.

Our measurement on 2026-11-14 shows the effect. The same Gulf-dialect chat of 117 words was 276 tokens on GLM 5.3 and 229 on Qwen3.8 27B. That is 276 / 117 x 100 = 235.9 versus 229 / 117 x 100 = 195.7 tokens per 100 words. The English translation of that chat cost 202 and 208 tokens, nearly the same.

## On BroadNet Inference

You do not choose a tokenizer on BroadNet; each model uses its own. Compare counts for your real text in the [Arabic token counter](https://broadrouter.com/tools/arabic-token-counter/) before you pick a model for Arabic traffic, and read [tokens per word in Arabic and English](https://broadrouter.com/blog/tokens-per-word-arabic-english/) for the full table. Remember that a [token](https://broadrouter.com/glossary/token/) count, not a word count, sets the price.

## Questions

### Who invented BPE for language models?

BPE began as a data compression algorithm. Sennrich, Haddow and Birch adapted it in 2015 to split rare words into subword units for neural machine translation, and LLM tokenizers grew from that work.

### What is byte-level BPE?

A BPE tokenizer whose base vocabulary is the 256 possible byte values instead of characters, so any text, in any script, can be encoded without an unknown token. GPT-2 used it with 50,257 entries.

### Why does BPE produce more tokens for Arabic than English?

Merges are learned from training data. If a tokenizer saw less Arabic, fewer Arabic pairs were merged, and Arabic words break into more pieces.

## See also

-   [Tokenizer](https://broadrouter.com/glossary/tokenizer/) (term)
-   [Vocabulary size](https://broadrouter.com/glossary/vocabulary-size/) (term)
-   [Token](https://broadrouter.com/glossary/token/) (term)
-   [Context window](https://broadrouter.com/glossary/context-window/) (term)
-   [Tokens per word in Arabic vs English: why Arabic costs more](https://broadrouter.com/blog/tokens-per-word-arabic-english/) (article)
-   [How to count tokens before you send an LLM request](https://broadrouter.com/blog/count-tokens-before-sending/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-bpe&utm_content=page-cta) ·

**Sources**

-   [Sennrich, Haddow and Birch, Neural Machine Translation of Rare Words with Subword Units (arXiv 1508.07909) (checked 2026-11-16)](https://arxiv.org/abs/1508.07909)
-   [Hugging Face Transformers, tokenization algorithms (BPE and byte-level BPE) (checked 2026-11-16)](https://huggingface.co/docs/transformers/tokenizer_summary)
