BroadRouter

Setting max_tokens for reasoning models: a budgeting method

By the BroadNet Inference team · 6 min read · Published

On a reasoning model, max_tokens caps thinking and answer together, so a tight limit can end the reply before any visible text. Measure total completion tokens per task type, set max_tokens to about the 95th percentile plus 20%, keep prompt plus max_tokens inside the context window, and retry cut-off replies with a doubled budget up to a hard cap.

A reasoning model thinks before it answers, and on most OpenAI-compatible APIs both come out of one budget: max_tokens. Set it the way you would for a non-reasoning model, a few hundred tokens for a short answer, and some replies come back empty with finish_reason: "length". The quick fix is in why a reasoning model returns an empty answer. This post is the method behind a budget you can trust: measure, size from percentiles, respect the context window, and handle the replies that still get cut off.

For what reasoning costs and how to reduce it, see reasoning tokens: why you pay for more output than you see.

Why does the budget run out before the answer?

The model generates in order: first its reasoning, then the visible answer. Every token of both counts toward max_tokens. BroadNet’s docs say this for DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B: “the reasoning counts toward max_tokens”, and with a small limit “the answer comes back empty”. OpenAI’s Chat Completions reference defines max_completion_tokens the same way, as an upper bound “including visible output tokens and reasoning tokens”.

So the question to ask isn’t “how long is my answer?” It is “how long are reasoning and answer together, on this task, at this effort?” That varies more than people expect. OpenAI’s guide says its models may generate “anywhere from a few hundred to tens of thousands of reasoning tokens”, depending on the problem.

Does a bigger max_tokens cost more?

No. This is the most common reason budgets are set too tight. max_tokens is a ceiling. You are billed for the tokens actually generated. A reply that needs 1,800 tokens costs the same with a ceiling of 4,000 or 40,000.

What a tight ceiling does cost:

Ceiling vs need What you pay What you get
Ceiling above need Tokens generated The answer
Ceiling below need The full ceiling, all of it reasoning Nothing usable, plus a retry

Take an illustration. Suppose a quarter of your calls need more than 1,000 tokens. With max_tokens=1000, each of those calls bills 1,000 output tokens of thinking, returns nothing and is retried. That adds 250 wasted output tokens per call on average across all calls, before the retries themselves, plus the latency of a second request. With the ceiling at 4,000, nothing is wasted. On Qwen3.8 27B, output costs $2.17 per 1M tokens, so waste adds up at volume.

The real risk of a high ceiling is a runaway reply: a model that loops or over-thinks and generates far more than needed. A hard cap bounds that. Pick the cap from data, as below.

How do you measure what a task needs?

Run real prompts for one task type with a generous ceiling, and record the completion tokens and the reasoning share. When the model reports it, usage.completion_tokens_details.reasoning_tokens gives the reasoning part; BroadNet’s docs list that field.

import os
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
MODEL = "broadnet/qwen3.8-27b"

def measure(prompts, ceiling=16_000, **kw):
    """Run real prompts with a generous ceiling and return per-call token counts."""
    rows = []
    for p in prompts:
        r = client.chat.completions.create(model=MODEL, max_tokens=ceiling,
                                           messages=[{"role": "user", "content": p}], **kw)
        u, d = r.usage, r.usage.completion_tokens_details
        reasoning = (d.reasoning_tokens or 0) if d else 0
        rows.append({"reasoning": reasoning, "answer": u.completion_tokens - reasoning,
                     "finish": r.choices[0].finish_reason})
    return rows

def pct(values, q):
    values = sorted(values)
    return values[min(len(values) - 1, int(q * len(values)))]

def recommend(rows, margin=1.2):
    total = [r["reasoning"] + r["answer"] for r in rows]
    return {"p50": pct(total, 0.5), "p95": pct(total, 0.95), "max": max(total),
            "max_tokens": int(pct(total, 0.95) * margin),
            "cut_off": sum(r["finish"] == "length" for r in rows)}

# print(recommend(measure(my_prompts)))                       # default effort
# print(recommend(measure(my_prompts, reasoning_effort="low")))

Rules for a number you can trust:

  1. One task type per run. Classification, extraction, code review and planning have very different distributions. One global max_tokens either wastes money on retries or lets runaway replies through.
  2. 50 or more real prompts. The 95th percentile of ten samples is just the largest one.
  3. Run it at each effort you use. reasoning_effort="low" changes the distribution. BroadNet documents low; test other values before you rely on them.
  4. If cut_off is not zero, raise the ceiling and run again. A cut-off sample hides its true length.
  5. Rerun when the model or prompt changes. A new model version can think more or less.

Then set max_tokens per task from recommend(). The 95th percentile plus 20% leaves most calls well under the ceiling and sends only the rare long one to the retry path below. The free reasoning token budget tool does this arithmetic in your browser once you have your numbers.

How does the context window limit the budget?

The context window holds the prompt and the answer together. BroadNet’s price feed defines context_length as “the most tokens one request can hold, prompt and answer together”. So:

max_tokens ≤ context length − prompt tokens

Model Context Room for max_tokens with a 200,000-token prompt
GLM 5.3 1,048,576 848,576
DeepSeek V4.1 Flash 1,048,576 848,576
Qwen3.8 27B 262,144 62,144

This matters for long-context work. An agent that has grown its history to 250,000 tokens on Qwen3.8 27B has only about 12,000 tokens left for thinking and answer. Long-context tasks also tend to need more reasoning, so check the remaining room at the moment of the call, not once at startup. Context window and the context window calculator cover the fit.

What should you do when a reply is still cut off?

Check finish_reason on every call. If it is length, the answer is missing or incomplete, even when content has some text. Don’t parse it as if it were complete; a truncated JSON object is a common source of downstream errors (structured output).

Retry with a doubled budget, inside the room the context allows, up to a hard cap:

import json, urllib.request

FEED = json.load(urllib.request.urlopen("https://inference.broadnet.ai/pricing.json"))
CONTEXT = {m["id"]: m.get("context_length") for m in FEED["models"]}

def ask(model, messages, budget=4_000, hard_cap=32_000, **kw):
    """Retry a cut-off reply with a doubled budget, never past the cap or the context window."""
    while True:
        r = client.chat.completions.create(model=model, messages=messages,
                                           max_tokens=budget, **kw)
        if r.choices[0].finish_reason != "length":
            return r
        room = CONTEXT[model] - r.usage.prompt_tokens        # context holds prompt + answer
        nxt = min(budget * 2, hard_cap, room)
        if nxt <= budget:
            raise RuntimeError(f"still cut off at max_tokens={budget}")
        print(f"cut off at {budget}, retrying with {nxt}")
        budget = nxt

Two alternatives to doubling, depending on the task:

  • Retry at reasoning_effort="low" with the same budget. For simple tasks the model often over-thinks, and less effort fixes it without a bigger ceiling.
  • Split the task. If a reply keeps hitting a high cap, the request is usually asking for too much at once.

Log every cut-off with the task type. A rising rate means the prompt, the model or the input data changed, and the budget needs measuring again. Retries are new inferences and are billed again, so keep the attempts bounded, as in long-running agents: timeouts and retries.

How does streaming change the picture?

With stream=True, the reasoning happens before the first visible token, so a reasoning model can look stalled at the start. In a chat UI, show a “thinking” state until content arrives, and size client timeouts for the thinking time, not just the answer.

Some servers also stream the reasoning text itself, in a separate field. The name is not standard: DeepSeek’s own API uses reasoning_content, and vLLM renamed its field from reasoning_content to reasoning, warning that old client code “could silently read an empty” field. BroadNet’s docs don’t document a reasoning text field, so don’t build features on one without checking the raw response for your model. If you do read it, read it defensively:

for chunk in stream:
    if chunk.choices:
        delta = chunk.choices[0].delta
        thinking = getattr(delta, "reasoning", None) or getattr(delta, "reasoning_content", None)
        # log or display `thinking` only if your provider actually sends it
        if delta.content:
            print(delta.content, end="")
    elif chunk.usage:                                  # last chunk, with include_usage
        print("\nreasoning tokens:", getattr(chunk.usage.completion_tokens_details, "reasoning_tokens", None))

The usage in the last chunk needs stream_options={"include_usage": True}, which BroadNet documents. More on streams in the streaming guide and SSE explained.

Which parameter name should you send?

API Parameter Includes reasoning
OpenAI Chat Completions max_completion_tokens (max_tokens deprecated) Yes
OpenAI Responses max_output_tokens Yes
BroadNet Inference (Chat Completions) max_tokens, as documented Yes
vLLM servers max_tokens, plus optional thinking_token_budget Yes

When moving code between providers, send the name the provider documents, and confirm with one test call that the limit takes effect. A limit silently ignored looks fine until a runaway reply arrives. The migration checklist covers the other parameters that change.

Where BroadNet Inference fits

BroadNet’s three reasoning models report reasoning tokens in usage when the model provides them, so the measurement above works as written. DeepSeek V4.1 Flash and GLM 5.3 give the most room, with 1,048,576 tokens of context. Qwen3.8 27B has 262,144. If you don’t need reasoning for a task, Gemma 4 VL 26B is not listed as a reasoning model and avoids the budget question altogether. On a flat DeepSeek V4.1 Flash plan, thinking tokens carry no per-token charge, but cut-off replies still cost time, so the same budgeting applies. BroadNet doesn’t document a separate reasoning cap; if you need one, use a provider or self-hosted server that offers it.

Questions

Why does my reasoning model return an empty response?

The model used the whole max_tokens budget on hidden reasoning before it wrote the answer. finish_reason is then "length". Raise max_tokens, lower reasoning_effort, or both. A short guide with the quick fix is linked from this post.

What max_tokens should I set for a reasoning model?

Measure first. Run 50 or more real prompts with a generous ceiling, take the 95th percentile of completion_tokens for that task, and add about 20%. BroadNet's docs give 1,000 tokens or more as a floor; OpenAI suggests reserving 25,000 when you start out on its models.

Does a higher max_tokens cost more?

Not by itself. You are billed for the tokens the model generates, not for the ceiling. A higher ceiling costs more only when a reply would otherwise have been cut off, and then a cut-off reply would have cost you a retry anyway.

Is max_completion_tokens the same as max_tokens?

On OpenAI, max_completion_tokens replaces the deprecated max_tokens and explicitly includes reasoning tokens. Many OpenAI-compatible providers document max_tokens instead. BroadNet's docs use max_tokens and say reasoning counts toward it.

Can I cap reasoning separately from the answer?

Only where the server supports it. vLLM has a thinking_token_budget parameter, and some providers have their own. BroadNet documents reasoning_effort set to low, not a separate reasoning cap, so budget the two together.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity