Tokens per second tells you how fast one answer streams. Throughput tells you how much a whole system gets done.
How it works
NVIDIA’s benchmarking guide separates the two. Total TPS per system is “total output token throughput across all simultaneous requests”: all output tokens divided by the time from the first request to the last response. TPS per user is one request’s output length divided by its end-to-end time. Requests per second (RPS) counts completed requests instead of tokens.
The key line: “As the number of concurrent requests increases, total system TPS increases while TPS per user decreases.” Total TPS climbs until the GPU is saturated, and past that point it can fall.
On the server, the limit is usually memory. The PagedAttention paper explains that serving at high throughput “requires batching sufficiently many requests at a time”, and the KV cache of each request decides how many fit. Better memory management let vLLM reach 2 to 4 times the throughput of earlier systems at the same latency.
Why it matters
For an API caller, throughput is about your side of the pipe: how many documents you can process per hour, or how many agents can work at once. It follows a simple rule:
your throughput ≈ concurrent requests × per-stream tokens per second
That holds only while the provider keeps per-stream speed steady as you add load. Measure it; do not assume it.
Throughput is capped by your concurrency limit and your rate limits, not by the model alone. If a job is slow, check whether requests run one after another when they could run in parallel.
On BroadNet Inference
BroadNet publishes limits per key, not a system throughput figure. Its plans page gives a typical speed of about 100 tokens per second for DeepSeek V4.1 Flash, the plan model; it publishes no speed for other models. As an illustration, if each stream ran at that figure:
| Option | Parallel requests | Upper bound if speed held |
|---|---|---|
| Pay-as-you-go key | 4 (and 30 requests per minute) | 4 streams × about 100 tokens per second each |
| Hourly key (Gemma 4 VL 26B) | 1 stream, 300,000 tokens per minute | not published; measure it |
| Flat plan session | 1 request at a time | about 100 tokens per second |
To scale a flat plan, you add sessions, one per agent working in parallel; see unlimited LLM API sessions and running parallel coding agents. That speed is BroadNet’s stated typical figure for DeepSeek V4.1 Flash, not a guarantee for other models or under load.