BroadRouter

LLM latency: TTFT vs tokens per second vs end-to-end time

By the BroadNet Inference team · 4 min read · Published · Updated

Time to first token (TTFT) is the wait before the first output token. Tokens per second is how fast the answer then streams. End-to-end latency is the total: TTFT plus generation time, roughly output tokens divided by per-user speed. Prompt length and queueing drive TTFT, model size and load drive speed, and answer length drives the total. Measure all three at p50 and p95.

“The API is slow” can mean three different things, and each has a different fix. A chat user waits for the first word. An agent waits for the whole tool call. A batch job cares about tokens per hour. This post defines the three numbers, shows how to measure them from your own code, and maps each fix to the number it actually moves.

What are TTFT, tokens per second and end-to-end latency?

NVIDIA’s benchmarking guide gives clean definitions, and they match what you can observe from a client:

Metric What it measures Who feels it Driven by
Time to first token (TTFT) Request sent → first token received Chat users Network, queueing, prompt length (prefill), cache hits
Inter-token latency (ITL) Average gap between tokens Readers of a streamed answer Model size, hardware, server load
Tokens per second per user Output tokens after the first ÷ generation time (first token to last); equals 1 ÷ average ITL Readers, agents Same as ITL
End-to-end latency Request sent → last token received Agents, APIs, batch TTFT + generation time

The formula that ties them together:

end-to-end ≈ TTFT + (output tokens − 1) × ITL ≈ TTFT + output tokens ÷ per-user tokens per second

Per-user speed is not the same as server throughput, the total tokens a server produces across all users; a provider can raise one while the other falls.

Take the typical speed BroadNet publishes for its plan model, about 100 tokens per second. Generation time scales with answer length, so a 4,000-token code file takes about eighty times as long to generate as a 50-token classification, while both wait the same TTFT. For short answers TTFT is most of the wait; for long ones, speed and length are.

Why do reasoning models need a fourth number?

Reasoning models such as DeepSeek V4.1 Flash think before they answer, and the thinking is output tokens too. If the model spends 800 tokens reasoning at about 100 tokens per second, the user waits for all 800 to be generated before the first visible word, even if TTFT on the wire looks fast. So measure time to first content token separately. On BroadNet Inference, reasoning_effort: "low" shortens the thinking for simple steps; the reasoning guide explains the trade-off.

How do you measure latency from the client?

Time it where your users are, with streaming on. Server-side timers miss the network and any proxy.

import os, time, statistics
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])

def timed(model, messages, max_tokens=1500):
    t0 = time.perf_counter()
    first = first_content = None
    usage = None
    stream = client.chat.completions.create(
        model=model, messages=messages, max_tokens=max_tokens,
        stream=True, stream_options={"include_usage": True},
    )
    for chunk in stream:
        now = time.perf_counter()
        if chunk.choices:
            if first is None:
                first = now
            if chunk.choices[0].delta.content and first_content is None:
                first_content = now
        if chunk.usage:
            usage = chunk.usage
    end = time.perf_counter()
    out = usage.completion_tokens if usage else 0
    return {
        "ttft": first - t0,
        "ttfct": (first_content or end) - t0,
        "e2e": end - t0,
        "tps": (out - 1) / (end - first) if out > 1 and end > first else 0,
        "prompt_tokens": usage.prompt_tokens if usage else None,
    }

runs = [timed("broadnet/gemma-4-26b-a4b", [{"role": "user", "content": "Summarise SSE in 100 words."}]) for _ in range(50)]
for k in ("ttft", "ttfct", "e2e", "tps"):
    xs = sorted(r[k] for r in runs)
    print(k, "p50", round(statistics.median(xs), 3), "p95", round(xs[int(len(xs) * 0.95) - 1], 3))

Five rules for a fair benchmark:

  1. Fix the prompt length and report it. TTFT at 500 prompt tokens and at 200,000 are different numbers. The token counter tells you how long a test prompt is before you send it.
  2. Fix the output length with max_tokens and a prompt that fills it, so tokens per second is comparable.
  3. Report p50 and p95, not the mean. Tail latency is what users remember.
  4. Spread runs over time. Load changes during the day.
  5. Warm up first. The first call pays for DNS, TLS and connection setup; reuse one client.

What actually improves each number?

To improve Do this Why
TTFT Shorten or cache the prompt prefix Prefill time grows with prompt length; cached prefixes skip work
TTFT Keep connections warm, use a nearby region Removes setup and round trips
Time to first content token Lower reasoning effort, or use a non-reasoning model for simple steps Less hidden thinking before the answer
Tokens per second Use a smaller or faster model for the step Decode speed depends on model size and serving
End-to-end Ask for shorter output (schemas, word limits) Output length multiplies ITL
End-to-end for many calls Run independent calls in parallel within your concurrency limit Wall-clock time drops even though each call is the same
Perceived latency Stream to the user They read while the model writes

Output length is the most underrated lever. Asking for “JSON with these four fields” instead of “explain your answer” can cut generation time by an order of magnitude with no infrastructure work. Structured output helps here.

The cache lever is covered in prompt structure for caching. The parallelism lever runs into rate limits quickly; see handling 429 errors.

How does prompt length change the trade-off?

TTFT grows with the prompt because the model must process every input token before it can write. That is the hidden cost of very long prompts and part of the long context vs RAG decision: a 1M-token prompt answers questions RAG cannot, but it starts slower. If interactive speed matters, keep the hot path short and send the big context only when the question needs it.

Where BroadNet Inference fits

BroadNet publishes a typical speed of about 100 tokens per second for its plan model and streams with the standard SSE format, so the code above works unchanged. It does not publish TTFT figures or latency SLAs, so measure on your own prompts before you commit; a $1 Meteor hour is enough for a 50-run benchmark, and the starting credit covers one on Gemma 4 VL 26B. If you need guaranteed sub-second latency at large scale, test carefully: a pay-as-you-go key allows 4 requests in flight, which suits apps and agents more than high-volume real-time serving. Models and their prices are on the models and pricing pages.

Questions

What is a good time to first token for an LLM API?

For chat, under about one second on short prompts feels responsive. Long prompts and reasoning models take longer by design. Compare TTFT only at the same prompt length and model.

Is tokens per second the same as throughput?

Not quite. Per-user tokens per second is how fast one answer streams. Throughput is the total tokens a server produces across all users. A provider can raise total throughput while each user's speed drops.

Why is TTFT so high on a reasoning model?

The model generates hidden reasoning tokens before the visible answer. The first token may arrive fast, but the first content token waits until thinking is done. Lower reasoning_effort for simple tasks.

Does streaming reduce latency?

It does not shorten end-to-end time. It lets the user read the answer while it is still being written, so perceived latency drops to roughly the TTFT.

How many requests do I need for a reliable latency benchmark?

Enough to see the tail. Fifty to a hundred requests per prompt size gives a usable p95. Spread them across the day, because load changes.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity