---
title: "What is LLM latency? End-to-end response time explained"
description: "LLM latency is how long a request takes, from sending the prompt to the last token. Its parts (TTFT, inter-token latency), percentiles and what changes it."
url: https://broadrouter.com/glossary/latency/
lang: en
updated: 2026-11-16
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Latency

Updated 2026-11-16

Latency, for a language model API, is the time a request takes from being sent to the final output token being received. It is the sum of time to first token and generation time, and is usually reported as percentiles such as p50 and p95 rather than as an average.

Also called: LLM latency, end-to-end latency, response time, e2e latency, inter-token latency

Latency is the number users feel. It is also the one that most often surprises teams moving from a demo to production.

## How it works

NVIDIA’s metrics guide gives the core formula for one request:

```text
e2e_latency = TTFT + generation_time
ITL = (e2e_latency - TTFT) / (output_tokens - 1)
```

[Time to first token](https://broadrouter.com/glossary/time-to-first-token/) covers network, queueing and reading the prompt. Generation time is everything after the first token: (output tokens − 1) × inter-token latency (ITL), or roughly output tokens × ITL for a long reply. End-to-end latency, NVIDIA says, “includes queueing, batching, and network latency.”

A worked example with round numbers: suppose a model streams 50 output tokens a second. A 600-token reply takes 600 ÷ 50 = 12 seconds to generate. Add one second of TTFT and the request takes about 13 seconds. Halve the reply and you save 6 seconds; halve TTFT and you save half a second. Here, output length dominates.

For an application, a request is rarely alone. Latency at the user’s screen also includes retries after errors, time waiting for a [concurrency](https://broadrouter.com/glossary/concurrency/) slot, tool calls between model turns, and your own code.

## Why it matters

OpenAI’s latency guide lists seven principles: process tokens faster, generate fewer tokens, use fewer input tokens, make fewer requests, parallelize, make your users wait less, and don’t default to an LLM. Most of them are design choices, not provider choices.

Report percentiles. A p50 of 4 seconds with a p95 of 30 seconds means one user in twenty waits half a minute or longer. Averages hide that tail, and timeouts are set by it.

Latency and [throughput](https://broadrouter.com/glossary/throughput/) pull against each other. NVIDIA notes that as concurrent requests grow, total system tokens per second rises while tokens per second per user falls. A busy provider can serve more people and still feel slower to each.

## On BroadNet Inference

BroadNet’s plans page gives a typical speed of about 100 tokens per second. Its docs do not publish TTFT or percentile latency figures. A pay-as-you-go key allows 4 requests in flight; the docs list a concurrency limit among the causes of HTTP 429, and every retry after one adds latency on your side. Long agent runs should set timeouts and bounded retries; see [timeouts and retries for long-running agents](https://broadrouter.com/blog/long-running-agents-timeouts-retries/). For measuring each part from the client, see the [LLM latency guide](https://broadrouter.com/blog/llm-latency-ttft-throughput/).

## Questions

### What is the difference between latency and throughput?

Latency is how long one request takes. Throughput is how much work a system finishes per second across all requests. A server can raise throughput by batching more requests while each one gets slower.

### What is inter-token latency?

The average gap between output tokens after the first one, also called time per output token. NVIDIA defines it as end-to-end latency minus TTFT, divided by output tokens minus one.

### How can I reduce LLM latency?

Generate fewer output tokens, send shorter prompts, stream the reply so users see text sooner, and run independent calls in parallel. OpenAI's latency guide lists these among its seven principles.

## See also

-   [Time to first token (TTFT)](https://broadrouter.com/glossary/time-to-first-token/) (term)
-   [Throughput](https://broadrouter.com/glossary/throughput/) (term)
-   [Tokens per second](https://broadrouter.com/glossary/tokens-per-second/) (term)
-   [Concurrency](https://broadrouter.com/glossary/concurrency/) (term)
-   [Rate limit](https://broadrouter.com/glossary/rate-limit/) (term)
-   [Stream chat completions and handle dropped connections](https://broadrouter.com/guides/streaming-chat-completions/) (guide)
-   [BroadNet Inference error codes and when to retry](https://broadrouter.com/guides/errors-and-retries/) (guide)
-   [LLM latency: TTFT vs tokens per second vs end-to-end time](https://broadrouter.com/blog/llm-latency-ttft-throughput/) (article)
-   [Long-running agents: timeouts, retries and checkpoints that hold](https://broadrouter.com/blog/long-running-agents-timeouts-retries/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-latency&utm_content=page-cta) ·

**Sources**

-   [NVIDIA NIM, LLM benchmarking metrics (end-to-end latency, inter-token latency)](https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html)
-   [OpenAI, latency optimization guide (seven principles)](https://developers.openai.com/api/docs/guides/latency-optimization)
-   [BroadNet Inference plans (typical speed)](https://inference.broadnet.ai/plans?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-latency&utm_content=sources)
