BroadRouter
Get started

Latency

Updated

Latency, for a language model API, is the time a request takes from being sent to the final output token being received. It is the sum of time to first token and generation time, and is usually reported as percentiles such as p50 and p95 rather than as an average.

Also called: LLM latency, end-to-end latency, response time, e2e latency, inter-token latency

Latency is the number users feel. It is also the one that most often surprises teams moving from a demo to production.

How it works

NVIDIA’s metrics guide gives the core formula for one request:

e2e_latency = TTFT + generation_time
ITL = (e2e_latency - TTFT) / (output_tokens - 1)

Time to first token covers network, queueing and reading the prompt. Generation time is everything after the first token: (output tokens − 1) × inter-token latency (ITL), or roughly output tokens × ITL for a long reply. End-to-end latency, NVIDIA says, “includes queueing, batching, and network latency.”

A worked example with round numbers: suppose a model streams 50 output tokens a second. A 600-token reply takes 600 ÷ 50 = 12 seconds to generate. Add one second of TTFT and the request takes about 13 seconds. Halve the reply and you save 6 seconds; halve TTFT and you save half a second. Here, output length dominates.

For an application, a request is rarely alone. Latency at the user’s screen also includes retries after errors, time waiting for a concurrency slot, tool calls between model turns, and your own code.

Why it matters

OpenAI’s latency guide lists seven principles: process tokens faster, generate fewer tokens, use fewer input tokens, make fewer requests, parallelize, make your users wait less, and don’t default to an LLM. Most of them are design choices, not provider choices.

Report percentiles. A p50 of 4 seconds with a p95 of 30 seconds means one user in twenty waits half a minute or longer. Averages hide that tail, and timeouts are set by it.

Latency and throughput pull against each other. NVIDIA notes that as concurrent requests grow, total system tokens per second rises while tokens per second per user falls. A busy provider can serve more people and still feel slower to each.

On BroadNet Inference

BroadNet’s plans page gives a typical speed of about 100 tokens per second. Its docs do not publish TTFT or percentile latency figures. A pay-as-you-go key allows 4 requests in flight; the docs list a concurrency limit among the causes of HTTP 429, and every retry after one adds latency on your side. Long agent runs should set timeouts and bounded retries; see timeouts and retries for long-running agents. For measuring each part from the client, see the LLM latency guide.

Questions

What is the difference between latency and throughput?

Latency is how long one request takes. Throughput is how much work a system finishes per second across all requests. A server can raise throughput by batching more requests while each one gets slower.

What is inter-token latency?

The average gap between output tokens after the first one, also called time per output token. NVIDIA defines it as end-to-end latency minus TTFT, divided by output tokens minus one.

How can I reduce LLM latency?

Generate fewer output tokens, send shorter prompts, stream the reply so users see text sooner, and run independent calls in parallel. OpenAI's latency guide lists these among its seven principles.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity