---
title: "What is a tokenizer? How LLMs split text into tokens"
description: "A tokenizer turns text into the token IDs an LLM reads. How it works and why the same text gives different token counts per model, with measured Arabic data."
url: https://broadrouter.com/glossary/tokenizer/
lang: en
updated: 2026-11-16
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Tokenizer

Updated 2026-11-16

A tokenizer is the component that converts text into the sequence of integer token IDs a language model reads, and converts generated IDs back into text. Each model ships its own tokenizer, so the same text can produce a different number of tokens on different models.

Also called: LLM tokenizer, tokenizer.json, subword tokenizer, text tokenizer

A tokenizer decides how many [tokens](https://broadrouter.com/glossary/token/) your text becomes, and so how much of the context window and the bill it uses.

## How it works

A tokenizer has two parts: a fixed vocabulary that maps text pieces to integer IDs, and rules for splitting text into those pieces. Hugging Face’s documentation describes the common algorithms as subword methods: frequent words stay whole and rare words split into smaller known pieces, so “annoyingly” can become “annoying” + “ly”. Most current LLMs use [byte pair encoding](https://broadrouter.com/glossary/bpe/) or a variant.

Encoding runs in three steps:

1.  **Pre-tokenize.** Split on spaces, punctuation and digits.
2.  **Map pieces to IDs.** Apply the learned merges and look up each piece in the vocabulary.
3.  **Add special tokens.** The chat template wraps messages in role markers, which also count.

Decoding reverses step 2. For open-weight models, all of this lives in a `tokenizer.json` file in the model repo.

## Why it matters

Different tokenizers give different counts for the same text. We encoded the dataset’s texts with each model’s pinned tokenizer (English: Wikipedia and Project Gutenberg prose; Arabic: 2,009 FLORES-200 sentences plus three Wikipedia leads; measured 2026-11-15; tokens per 100 words = tokens / words x 100):

| Text | DeepSeek V4.1 Flash | GLM 5.3 | Qwen3.8 27B | Gemma 4 | GPT o200k |
| --- | --- | --- | --- | --- | --- |
| English prose, 1,183 words | 1,507 → 127.4 | 1,526 → 129.0 | 1,561 → 132.0 | 1,537 → 129.9 | 1,501 → 126.9 |
| Arabic MSA, 39,037 words | 86,690 → 222.1 | 95,230 → 243.9 | 71,424 → 183.0 | 79,120 → 202.7 | 73,593 → 188.5 |

English is close everywhere. Arabic is not: GLM 5.3 needs 95,230 / 71,424 = 1.33 times the tokens Qwen3.8 27B needs for the same text. On a per-token price, that gap is a cost gap, and it also changes how much Arabic fits in the [context window](https://broadrouter.com/glossary/context-window/). The Gemma column uses Google’s base Gemma 4 tokenizer; BroadNet’s build is based on that model, and we assume it keeps the tokenizer. The full per-model numbers, with CSV and JSON downloads, are in the [tokenizer efficiency dataset](https://broadrouter.com/data/tokenizer-efficiency/).

## On BroadNet Inference

BroadNet bills pay-as-you-go chat by [input and output tokens](https://broadrouter.com/glossary/input-and-output-tokens/) and reports the counts in each response’s `usage` object, so the authoritative number is the one the API returns. To estimate before you send, paste text into the [token counter](https://broadrouter.com/tools/token-counter/) or the [Arabic token counter](https://broadrouter.com/tools/arabic-token-counter/), which load the same pinned tokenizer files. The full measurement is in [tokens per word in Arabic and English](https://broadrouter.com/blog/tokens-per-word-arabic-english/).

## Common mistakes

-   Counting with one model’s tokenizer and billing on another. Use the tokenizer of the model you call.
-   Forgetting the chat template. Role markers and system prompts add tokens that plain-text counts miss.

## Questions

### Do all LLMs use the same tokenizer?

No. Each model family trains its own, with its own vocabulary. In our tokenizer dataset the same 39,037 words of Arabic were 71,424 tokens on Qwen3.8 27B and 95,230 on GLM 5.3.

### Where do I find a model's tokenizer?

For open-weight models it is the tokenizer.json file in the model's Hugging Face repo. Load it with the Hugging Face tokenizers library to count tokens exactly.

### Is a tokenizer the same as tokenization?

Tokenization is the process of splitting text into tokens. The tokenizer is the file and code that does it for one specific model.

## See also

-   [Token](https://broadrouter.com/glossary/token/) (term)
-   [Byte pair encoding (BPE)](https://broadrouter.com/glossary/bpe/) (term)
-   [Vocabulary size](https://broadrouter.com/glossary/vocabulary-size/) (term)
-   [Input and output tokens](https://broadrouter.com/glossary/input-and-output-tokens/) (term)
-   [Context window](https://broadrouter.com/glossary/context-window/) (term)
-   [Chat template](https://broadrouter.com/glossary/chat-template/) (term)
-   [Unicode normalization](https://broadrouter.com/glossary/unicode-normalization/) (term)
-   [Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/) (guide)
-   [Tokens per word in Arabic vs English: why Arabic costs more](https://broadrouter.com/blog/tokens-per-word-arabic-english/) (article)
-   [How to count tokens before you send an LLM request](https://broadrouter.com/blog/count-tokens-before-sending/) (article)
-   [Arabic prompt engineering: register, RTL, numerals and mixed text](https://broadrouter.com/blog/arabic-prompt-engineering/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-tokenizer&utm_content=page-cta) ·

**Sources**

-   [Hugging Face Transformers, tokenization algorithms (checked 2026-11-16)](https://huggingface.co/docs/transformers/tokenizer_summary)
-   [Hugging Face model repo, Qwen/Qwen3.8-27B (checked 2026-11-16)](https://huggingface.co/Qwen/Qwen3.8-27B)
-   [Hugging Face model repo, deepseek-ai/DeepSeek-V4.1-Flash (checked 2026-11-16)](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash)
-   [Hugging Face model repo, google/gemma-4-26B-A4B-it (checked 2026-11-16)](https://huggingface.co/google/gemma-4-26B-A4B-it)
