BroadRouter

Vocabulary size

Updated

Vocabulary size is the number of distinct tokens in a language model's tokenizer, each with its own ID and embedding. A larger vocabulary can encode common text in fewer tokens, but it makes the embedding and output layers bigger, and it only helps for text the vocabulary was trained on.

Also called: vocab size, vocab_size, tokenizer vocabulary, token vocabulary

Vocabulary size tells you how many different pieces of text a model can represent with a single token.

How it works

A BPE tokenizer’s vocabulary is its base symbols plus every merge it learned. Hugging Face’s documentation gives the original GPT as an example: 478 base tokens + 40,000 merges = 40,478. The builder picks the target size before training.

We read the vocabularies from each model’s pinned tokenizer.json on Hugging Face on 2026-11-14:

Tokenizer Vocabulary entries Total IDs incl. special tokens
DeepSeek V4.1 Flash 128,000 129,280
GLM 5.3 154,820 154,856
Qwen3.8 27B 248,044 248,077
Gemma 4 26B A4B (Google base) 262,144 262,144
OpenAI o200k_base (reference) 199,998 200,019

The Gemma 4 model card states a 262K vocabulary, which matches. The Qwen3.8 card lists an embedding size of 248,320 “padded”, larger than the 248,077 IDs actually used.

Why it matters

For an API user, vocabulary size is an indirect hint about cost per word. What counts is which text the merges were trained on. Compare tokens per 100 words for the same 39,037 words of Modern Standard Arabic in our tokenizer dataset:

  • DeepSeek V4.1 Flash, 128,000 entries: 86,690 / 39,037 x 100 = 222.1
  • GLM 5.3, 154,820 entries: 95,230 / 39,037 x 100 = 243.9
  • Qwen3.8 27B, 248,044 entries: 71,424 / 39,037 x 100 = 183.0
  • Gemma 4, 262,144 entries: 79,120 / 39,037 x 100 = 202.7

GLM 5.3 has more entries than DeepSeek but uses more tokens on this text. Gemma has the most entries but is not the most compact. For English prose all five land between 127 and 132 tokens per 100 words, so vocabulary size barely matters there.

The rule: measure your own text with the model’s tokenizer rather than inferring cost from the vocabulary number.

On BroadNet Inference

BroadNet does not publish vocabulary sizes; the figures above come from the public model repos. Gemma’s row uses Google’s base tokenizer, which we assume BroadNet’s Gemma 4 VL 26B build keeps. To see the effect on your own prompts, use the token counter, and for model choice in Arabic read choosing an LLM for Arabic alongside the tokens per word study.

Questions

Does a bigger vocabulary mean fewer tokens?

Often, but not always. In our 2026-11-14 test Gemma 4 has the largest vocabulary (262,144) yet used 347 tokens for an Arabic text that Qwen3.8 27B, with 248,044 entries, encoded in 316.

Why do vocab_size and the tokenizer file disagree?

Model configs often pad the embedding table to a round number and reserve IDs for special tokens. Qwen3.8 27B lists a padded size of 248,320, while its tokenizer.json uses IDs 0 to 248,076 (IDs start at 0), so 248,077 in all.

Where do I find a model's vocabulary size?

In the model's config.json as vocab_size, or by loading its tokenizer.json and counting entries. The config value is often padded, so the tokenizer file gives the exact count of tokens in use.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity