BroadRouter

How to count tokens before you send an LLM request

By the BroadNet Inference team · 5 min read · Published · Updated

Count tokens with the tokenizer of the model you call when it is public; otherwise calibrate a characters-per-token ratio from the usage field of real requests, per language and content type. Add a 10 to 20 percent margin, then check that prompt tokens plus max_tokens fit the context window, leaving room for reasoning tokens on thinking models.

You need a token count before you send a request for three reasons: to stay inside the context window, to pick a sensible max_tokens, and to estimate cost. Exact counts come only from the model’s own tokenizer. Good estimates come from calibration. This post shows both, with real measurements.

Why do token counts differ between models?

A tokenizer splits text into pieces from a fixed vocabulary. Each model family trains its own. A larger vocabulary, or one trained on more of a given language, produces fewer tokens for that language. So “how many tokens is this prompt” has a different answer for each model.

We measured the same short customer-service message, translated into four languages, plus a small Python function and an invoice JSON object. The OpenAI counts come from tiktoken (cl100k_base, GPT-4 era, and o200k_base, GPT-4o era). The open-model counts come from the tokenizer files each family publishes on Hugging Face, counted without the chat template, on 14 November 2026.

Text (same meaning) Characters cl100k_base o200k_base DeepSeek V4.1 Flash GLM 5.3 Qwen3.8 27B Gemma 4
English, 31 words 173 37 37 37 38 39 39
Spanish 160 47 40 50 45 43 42
Arabic 142 102 46 55 58 48 49
Chinese 54 51 37 34 35 36 38
Python function 94 33 33 35 33 35 38
Invoice JSON 140 59 60 63 60 72 72

One sample per row, so treat the numbers as an illustration, not a benchmark. Three lessons hold anyway. The tokenizer generation matters: the Arabic text more than halved from cl100k_base to o200k_base. Open models differ from each other too: the same Arabic text ranged from 48 to 58 tokens, and the JSON from 60 to 72. And structured content is token-dense everywhere: the JSON ran at about 2 characters per token on every tokenizer.

You can repeat this on your own text with the free token counter, or the per-family counters for DeepSeek, Qwen and GPT. To see what an edit to a prompt saves, compare both versions in the prompt token diff. Why Arabic in particular costs more tokens, measured across longer texts, is covered in tokens per word in Arabic vs English.

How do you count tokens exactly?

If the model’s tokenizer is public, load it and count locally. For OpenAI models:

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
n = len(enc.encode(text))

For open-weight families, the tokenizer usually ships with the model on Hugging Face and loads with AutoTokenizer.from_pretrained(...). Two caveats: the chat template adds tokens for roles and separators, and an API provider’s deployed version may differ from the public files. BroadNet’s API reference does not document a token-counting endpoint or the tokenizer for each model, so the public tokenizer gives a close estimate and the exact number is usage.prompt_tokens after the call.

How do you estimate tokens when the tokenizer is not public?

Calibrate once per model, then estimate locally:

import os
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])

def prompt_tokens(model, text):
    r = client.chat.completions.create(
        model=model, max_tokens=1000,  # room for reasoning on thinking models
        messages=[{"role": "user", "content": text}],
    )
    return r.usage.prompt_tokens

def measure(model, samples: dict[str, list[str]]):
    base = prompt_tokens(model, "hi")  # chat-template overhead plus one short word
    ratios = {}
    for kind, texts in samples.items():
        chars = toks = 0
        for t in texts:
            chars += len(t)
            toks += prompt_tokens(model, t) - base
        ratios[kind] = chars / toks
    return ratios  # e.g. {"english": 4.1, "arabic": 2.9, "json": 2.3}

def estimate(text, kind, ratios, overhead=10):
    return int(len(text) / ratios[kind]) + overhead

Use 20 or more real samples per content type (your actual prompts, not lorem ipsum). The one-word baseline request removes the fixed chat-template overhead, which estimate adds back as overhead; set it to your measured baseline. Calibrate on BroadNet’s cheapest model only if it is the one you will call: Gemma 4 VL 26B ratios do not transfer to DeepSeek V4.1 Flash. Images are billed as input tokens too, and larger images use more, so calibrate them separately by resolution.

What margins should you use for max_tokens and context?

The window covers prompt and output. On reasoning models the output includes hidden thinking, reported as reasoning tokens. Budget like this:

  1. Prompt estimate = calibrated count × 1.15 (a 15 percent safety margin).
  2. Answer budget = the longest answer you accept.
  3. Reasoning budget = for thinking models, at least 1,000 tokens; BroadNet’s docs recommend at least 1,000 or reasoning_effort: "low", because a small limit can return an empty answer.
  4. max_tokens = answer budget + reasoning budget.
  5. Check: prompt estimate + max_tokens ≤ context window. If not, cut the prompt, not the answer.

Context windows on BroadNet Inference, from the published price file:

Model Context window Practical prompt ceiling with 16K output and 15% margin
DeepSeek V4.1 Flash 1,048,576 about 897,000
GLM 5.3 1,048,576 about 897,000
Qwen3.8 27B 262,144 about 213,000
Gemma 4 VL 26B 32,768 about 14,000

The ceiling column is (window − 16,384) ÷ 1.15, rounded down. Change 16,384 to your own max_tokens.

What goes wrong with token estimates?

  • Counting words instead of tokens. The 0.75-words-per-token rule is for English. Our 29-word Arabic sample was 46 to 102 tokens depending on the tokenizer.
  • Forgetting tools and system prompt. Tool schemas are JSON, which is token-dense. Count them as part of the prompt.
  • Assuming silent truncation. Some tools drop old messages without telling you. BroadNet says it does not trim your context, so check the size before you send rather than relying on the server to cut it. The context window calculator does the check for you.
  • Ignoring history growth. In a chat or agent loop, the prompt grows every turn. Re-check the budget each call, not once per session. Stable history also caches better; see prompt structure for caching.

If you work in Arabic, the tokenizer is one more reason to test models on your own text; choosing an LLM for Arabic covers the rest. To turn token counts into a monthly bill, see how to estimate LLM API cost before launch.

Where BroadNet Inference fits

BroadNet Inference returns prompt_tokens, completion_tokens and, when the model reports them, cached and reasoning token counts in every response, which is all you need to calibrate. It does not publish a counting endpoint or per-model tokenizers, so pre-send counts are estimates. Long-context work on the 1M-token models is covered in the context window guide, and prices per 1M tokens are on the pricing page.

Questions

How many tokens is a word?

In BroadRouter's tokenizer dataset, which pools five tokenizers (those of the four BroadNet chat models and OpenAI's o200k_base; the post's example table adds cl100k_base), 1,000 English words is about 1,290 tokens, or about 1.29 tokens per word. The common rule of thumb, 0.75 words per token, is close for English prose. Arabic, other non-Latin scripts and code use more tokens per word. Measure your own text with the model you call.

Can I use tiktoken to count tokens for DeepSeek, Qwen or Gemma?

Only as a rough estimate. tiktoken implements OpenAI encodings. Other model families ship their own tokenizers, often on Hugging Face, and the counts differ. Calibrate against the usage field.

Is there a free tool to count tokens for DeepSeek, Qwen, GLM or Gemma?

Yes. BroadRouter's free token counter runs the published tokenizers for these model families in your browser, so the text does not leave your machine. Treat the count as an estimate of what the API will report, because chat templates add a few tokens.

What happens if my prompt is longer than the context window?

Most APIs return an error. Some tools trim old messages silently. BroadNet does not trim your context, so check prompt plus max_tokens against the window before sending.

How much margin should I leave for max_tokens?

Estimate the longest answer you expect, add the reasoning budget for thinking models, and keep 10 to 20 percent of the window free for estimation error.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity