BroadRouter

Time to first token (TTFT)

Updated

Time to first token (TTFT) is the time from sending a request to a language model until the first output token arrives. It includes network time, queueing on the server and prefill, the step where the model reads the whole prompt, so it grows with prompt length and load.

Also called: TTFT, first-token latency, time to first content token, first token time

TTFT is the pause before anything happens. In a chat window it is the blank screen; in a voice bot it is dead air.

How it works

NVIDIA’s benchmarking guide breaks TTFT into three parts: “request queuing time, prefill time, and network latency.” Each has its own cause:

Part What drives it What you can change
Network Distance, TLS setup, proxies Reuse connections; call from a nearby region
Queueing How busy the server is Off-peak timing, a less loaded model or tier
Prefill Prompt length, model size Shorter prompts, stable cacheable prefixes

Prefill is usually the big one for long inputs. The model runs the whole prompt through every layer to build the KV cache before it can produce token one. NVIDIA notes that on a busy server one request’s prefill can overlap with another’s generation, so TTFT also moves with other people’s traffic.

Two definitions matter when you read numbers. NVIDIA’s AIPerf counts the first token only if it carries content. For streaming text in the Responses API, OpenAI’s guide times TTFT “until you receive the first nonempty response.output_text.delta”. With reasoning tokens, the first visible word can come much later than the first generated token, so say which one you mean.

Why it matters

TTFT sets how fast a product feels, more than raw speed does. OpenAI’s guide says “a long prompt can delay the first token, especially when much of the input is new”. This is where cached input tokens help. The same guide says prompt caching “can reuse an eligible exact prefix match” and suggests comparing TTFT with and without cache hits. A cached prefix skips part of prefill, so a repeated system prompt costs less and can start sooner.

For agents, TTFT is paid on every step. Twenty tool calls with a two-second TTFT add forty seconds of waiting before counting any output.

TTFT is one part of end-to-end latency. The rest is generation time, set by tokens per second.

On BroadNet Inference

BroadNet’s docs describe streaming with Server-Sent Events, ending with data: [DONE], which is how you observe the first token on the client. DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason before they answer, so time to first visible text includes thinking. BroadNet does not publish a TTFT figure. For a timing script and p50/p95 reporting, see LLM latency: TTFT vs tokens per second.

Questions

Why is TTFT slow with a long prompt?

The model has to read every prompt token and build its KV cache before it can write anything. NVIDIA's metrics guide says longer prompts increase TTFT for exactly this reason.

Does streaming reduce time to first token?

No. Streaming lets you see the first token as soon as it exists; without it you wait for the whole reply. The server-side work before that token is the same either way.

Does an empty first chunk count as the first token?

Not in most benchmarks. NVIDIA's AIPerf ignores initial responses with no content, and OpenAI times TTFT to the first non-empty text delta.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity