BroadRouter

LLM token counter

Updated

In short: A token is the unit an LLM reads and bills: about 4.6 characters, or 0.78 English words, on average in our open dataset. Each model family has its own tokenizer, so counts differ: Arabic and code vary most. Paste text for counts with each model's own tokenizer (Gemma's from its base model) and the cost per model.

Free, no sign-up, runs in your browser: nothing you type is sent anywhere.

Try:

… tokens

Tokens and cost of this text per model
ModelTokensChars per tokenAs inputAs output
DeepSeek V4.1 Flash
DeepSeek V4.1 Flash tokenizer
…………
GLM 5.3
GLM 5.3 tokenizer
…………
Qwen3.8 27B
Qwen3.8 27B tokenizer
…………
Gemma 4 VL 26B
Gemma 4 tokenizer (Google base model)
…………
GPT (o200k)
OpenAI o200k_base
…………

Cost = tokens × the model's published price per 1M input or output tokens. GPT is shown for reference only (no BroadNet price). Measured ratios for every tokenizer are in the tokenizer efficiency dataset. Rows marked ≈ are estimates from measured characters per token until you load that tokenizer; loading downloads its tokenizer.json from Hugging Face once and keeps it in your browser cache.

Create an account ($5 starting credit)

Per-model counters: DeepSeek token counter · GLM token counter · Qwen token counter · Gemma token counter · GPT token counter. Prices synced 2026-11-14.

Report a wrong price or number (email to info@broadnet.ai, with this page's link and your inputs filled in).

How it works

  1. Paste text, code or JSON into the box, or pick a sample. The count updates as you type.
  2. Pick which tokenizer to view. The text is shown split into tokens, with a colour per token; tick "Show token IDs" to see the numbers.
  3. Tick "Chat format" to add a system prompt and the template tokens a chat request adds around messages.
  4. Read the table: tokens, characters per token and the input and output cost for each model. Load the other real tokenizers to replace estimates with real counts, then copy the link or download CSV.

Which tokenizer each model uses

Every count on this page comes from a real tokenizer file, pinned to a specific commit so the numbers do not drift. We checked on 2026-11-15 that each repository is public, not gated, and that the browser library produces the same token IDs as Hugging Face's Rust tokenizers reference on every sample below.

ModelTokenizer sourceToken IDsContextDownload
DeepSeek V4.1 Flashdeepseek-ai/DeepSeek-V4.1-Flash
exact model tokenizer
129,2801,048,5762 MB
GLM 5.3zai-org/GLM-5.3
exact model tokenizer
154,8561,048,57620 MB
Qwen3.8 27BQwen/Qwen3.8-27B
exact model tokenizer
248,077262,14413 MB
BroadNet Gemma 4 VL 26Bgoogle/gemma-4-26B-A4B-it
base-model tokenizer
262,14432,76832 MB
GPT-4o, GPT-4.1, o-series, GPT-5 (reference)o200k_base via gpt-tokenizer
exact model tokenizer
200,019varies1.1 MB

Google's Gemma 4 26B A4B tokenizer. BroadNet's build is based on this model; we assume it keeps the base tokenizer.

Same text, different token counts

We ran each tokenizer on the same open-licensed texts (method below). The table shows tokens per 100 words; lower is cheaper for the same text.

SampleWordsDeepSeekGLMQwenGemmaGPT
English prose1183127129132130127
Arabic (Modern Standard)39037222244183203189
Gulf Arabic203232251221226205
Chinese (Simplified)45649112114111129148
Spanish52022155150139135135
Hindi51486293455215138164
Code: Python280211200215234203
JSON1706401411484494389

Method: Parallel text: FLORES-200 dev and devtest (2,009 aligned sentences per language, CC BY-SA 4.0, tarball pinned by SHA-256) for Modern Standard, Egyptian, North Levantine, Najdi and Mesopotamian Arabic, Hindi, Urdu, Chinese, French and Spanish against English, each sentence counted on its own; Tatoeba sentence pairs (CC BY 2.0 FR) for Gulf Arabic and Arabizi. Ratio = pooled tokens ÷ pooled tokens of the English side, with a 95% interval from a seeded bootstrap over sentences (10,000 resamples). Other texts: Wikipedia lead sections (CC BY-SA 4.0), Project Gutenberg (public domain) and Wikidata JSON (CC0), stored with source URL, licence and revision. Tokens: the official tokenizer file at a pinned Hugging Face commit, encoded with @huggingface/tokenizers with no special tokens; o200k_base via gpt-tokenizer; Kimi's tiktoken rank file with a tiktoken-compatible counter. Gated Meta and Google files were read from ungated mirrors proven byte-identical to the official file. Every new count matched the Rust tokenizers library or Python tiktoken. Words: word-like segments from Intl.Segmenter. Characters: Unicode code points. Measured 2026-11-15. Bold = fewest tokens. Download the counts as CSV or JSON from the tokenizer efficiency dataset.

English is nearly a tie. The spread opens up on other scripts: on Modern Standard Arabic the counts range from 183 to 244 tokens per 100 words. That is why a model with a lower price per million tokens can still cost more for your text. The Arabic token counter measures this for your own text.

The cost formula

input cost  = prompt tokens     × input price per 1M  / 1,000,000
output cost = completion tokens × output price per 1M / 1,000,000

Worked example: the English sample in the counter is 260 words and 299 tokens with the DeepSeek V4.1 Flash tokenizer. Sent as a prompt it costs $0.000052, or $0.0523 for 1,000 such requests. Cached input and reasoning tokens change the bill; the LLM API cost calculator handles both. See also input and output tokens.

What the chat format adds

An API request is not just your text. The server wraps each message in the model's chat template: role markers, special tokens and, for these reasoning models, a short reasoning-effort preamble. With the official templates' default settings a one-message request adds about 12 to 52 tokens depending on the model, plus a few per extra message. Tool definitions add far more. Tick "Chat format" to see the exact template tokens in the token view.

Questions

How do I count tokens for DeepSeek, Qwen, GLM or Gemma?

Paste your text above. The counter runs each model's own tokenizer.json, downloaded from the model's official Hugging Face repository, in your browser. OpenAI-only counters such as tiktoken give the wrong number for these models because each family has its own vocabulary.

Is the count the same as the API will bill?

For the text itself, yes for DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B, whose tokenizers come from the exact model repositories. A real chat request also adds template tokens around each message; tick "Chat format" to include them as the official chat template renders them. The usage.prompt_tokens field in the API response is always the final word.

Why does the same text have different token counts on different models?

Each model family trains its own tokenizer with its own vocabulary, from about 129,000 to 262,000 token IDs here. A vocabulary trained on more Arabic, Hindi or Chinese packs those languages into fewer tokens. In our dataset 100 words of Hindi take from 138 to 455 tokens depending on the tokenizer.

How many tokens is 1,000 words?

About 1,290 tokens for English prose, averaged over these five tokenizers (127 to 132 per 100 words, measured on 1,183 words of open-licensed prose). Code, JSON and Arabic need far more. See the tokens to words converter for every content type.

Why does a tokenizer need to download first?

A tokenizer is a vocabulary plus merge rules, stored in a tokenizer.json of 2 MB to 32 MB. The counter fetches one only when you use it and keeps it in your browser cache, so the next visit is instant. Until then the row shows an estimate marked ≈.

Do output and reasoning tokens use the same tokenizer?

Yes. The model writes with the same vocabulary, so this counter also tells you what a reply of that length costs as output. Reasoning tokens are billed as output too. What reasoning tokens are.

Go deeper

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity