Latency is the number users feel. It is also the one that most often surprises teams moving from a demo to production.
How it works
NVIDIA’s metrics guide gives the core formula for one request:
e2e_latency = TTFT + generation_time
ITL = (e2e_latency - TTFT) / (output_tokens - 1)
Time to first token covers network, queueing and reading the prompt. Generation time is everything after the first token: (output tokens − 1) × inter-token latency (ITL), or roughly output tokens × ITL for a long reply. End-to-end latency, NVIDIA says, “includes queueing, batching, and network latency.”
A worked example with round numbers: suppose a model streams 50 output tokens a second. A 600-token reply takes 600 ÷ 50 = 12 seconds to generate. Add one second of TTFT and the request takes about 13 seconds. Halve the reply and you save 6 seconds; halve TTFT and you save half a second. Here, output length dominates.
For an application, a request is rarely alone. Latency at the user’s screen also includes retries after errors, time waiting for a concurrency slot, tool calls between model turns, and your own code.
Why it matters
OpenAI’s latency guide lists seven principles: process tokens faster, generate fewer tokens, use fewer input tokens, make fewer requests, parallelize, make your users wait less, and don’t default to an LLM. Most of them are design choices, not provider choices.
Report percentiles. A p50 of 4 seconds with a p95 of 30 seconds means one user in twenty waits half a minute or longer. Averages hide that tail, and timeouts are set by it.
Latency and throughput pull against each other. NVIDIA notes that as concurrent requests grow, total system tokens per second rises while tokens per second per user falls. A busy provider can serve more people and still feel slower to each.
On BroadNet Inference
BroadNet’s plans page gives a typical speed of about 100 tokens per second. Its docs do not publish TTFT or percentile latency figures. A pay-as-you-go key allows 4 requests in flight; the docs list a concurrency limit among the causes of HTTP 429, and every retry after one adds latency on your side. Long agent runs should set timeouts and bounded retries; see timeouts and retries for long-running agents. For measuring each part from the client, see the LLM latency guide.