BroadRouter
Get started

Throughput

Updated

Throughput, in LLM serving, is the total amount of work a system completes per unit of time across all concurrent requests, usually measured as output tokens per second or requests per second. It differs from per-stream speed, which is how fast a single request receives its tokens.

Also called: system throughput, aggregate throughput, total tokens per second, requests per second, RPS

Tokens per second tells you how fast one answer streams. Throughput tells you how much a whole system gets done.

How it works

NVIDIA’s benchmarking guide separates the two. Total TPS per system is “total output token throughput across all simultaneous requests”: all output tokens divided by the time from the first request to the last response. TPS per user is one request’s output length divided by its end-to-end time. Requests per second (RPS) counts completed requests instead of tokens.

The key line: “As the number of concurrent requests increases, total system TPS increases while TPS per user decreases.” Total TPS climbs until the GPU is saturated, and past that point it can fall.

On the server, the limit is usually memory. The PagedAttention paper explains that serving at high throughput “requires batching sufficiently many requests at a time”, and the KV cache of each request decides how many fit. Better memory management let vLLM reach 2 to 4 times the throughput of earlier systems at the same latency.

Why it matters

For an API caller, throughput is about your side of the pipe: how many documents you can process per hour, or how many agents can work at once. It follows a simple rule:

your throughput ≈ concurrent requests × per-stream tokens per second

That holds only while the provider keeps per-stream speed steady as you add load. Measure it; do not assume it.

Throughput is capped by your concurrency limit and your rate limits, not by the model alone. If a job is slow, check whether requests run one after another when they could run in parallel.

On BroadNet Inference

BroadNet publishes limits per key, not a system throughput figure. Its plans page gives a typical speed of about 100 tokens per second for DeepSeek V4.1 Flash, the plan model; it publishes no speed for other models. As an illustration, if each stream ran at that figure:

Option Parallel requests Upper bound if speed held
Pay-as-you-go key 4 (and 30 requests per minute) 4 streams × about 100 tokens per second each
Hourly key (Gemma 4 VL 26B) 1 stream, 300,000 tokens per minute not published; measure it
Flat plan session 1 request at a time about 100 tokens per second

To scale a flat plan, you add sessions, one per agent working in parallel; see unlimited LLM API sessions and running parallel coding agents. That speed is BroadNet’s stated typical figure for DeepSeek V4.1 Flash, not a guarantee for other models or under load.

Questions

Is throughput the same as tokens per second?

Only at system level. Per-stream tokens per second is the speed one request sees. Throughput adds up all streams. NVIDIA calls these TPS per user and total TPS per system.

How do I get more throughput from an LLM API?

Run more requests in parallel, up to your concurrency and rate limits, and keep each request short. If each request already uses its slot fully, more slots, keys or sessions are the only way up.

Does higher throughput mean lower latency?

No. Throughput is total work per second across all requests; latency is how long one request takes. Serving more requests at once raises total throughput but can slow each request, which is why NVIDIA reports TPS per user and total TPS separately.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity