---
title: "What is LLM throughput? System-wide tokens per second"
description: "LLM throughput is the total work a system finishes per second across all concurrent requests, in tokens or requests per second. How it differs from per-stream speed."
url: https://broadrouter.com/glossary/throughput/
lang: en
updated: 2026-11-14
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Throughput

Updated 2026-11-14

Throughput, in LLM serving, is the total amount of work a system completes per unit of time across all concurrent requests, usually measured as output tokens per second or requests per second. It differs from per-stream speed, which is how fast a single request receives its tokens.

Also called: system throughput, aggregate throughput, total tokens per second, requests per second, RPS

[Tokens per second](https://broadrouter.com/glossary/tokens-per-second/) tells you how fast one answer streams. Throughput tells you how much a whole system gets done.

## How it works

NVIDIA’s benchmarking guide separates the two. **Total TPS per system** is “total output token throughput across all simultaneous requests”: all output tokens divided by the time from the first request to the last response. **TPS per user** is one request’s output length divided by its end-to-end time. **Requests per second (RPS)** counts completed requests instead of tokens.

The key line: “As the number of concurrent requests increases, total system TPS increases while TPS per user decreases.” Total TPS climbs until the GPU is saturated, and past that point it can fall.

On the server, the limit is usually memory. The PagedAttention paper explains that serving at high throughput “requires batching sufficiently many requests at a time”, and the [KV cache](https://broadrouter.com/glossary/kv-cache/) of each request decides how many fit. Better memory management let vLLM reach 2 to 4 times the throughput of earlier systems at the same latency.

## Why it matters

For an API caller, throughput is about your side of the pipe: how many documents you can process per hour, or how many agents can work at once. It follows a simple rule:

```text
your throughput ≈ concurrent requests × per-stream tokens per second
```

That holds only while the provider keeps per-stream speed steady as you add load. Measure it; do not assume it.

Throughput is capped by your [concurrency](https://broadrouter.com/glossary/concurrency/) limit and your rate limits, not by the model alone. If a job is slow, check whether requests run one after another when they could run in parallel.

## On BroadNet Inference

BroadNet publishes limits per key, not a system throughput figure. Its plans page gives a typical speed of about 100 tokens per second for DeepSeek V4.1 Flash, the plan model; it publishes no speed for other models. As an illustration, if each stream ran at that figure:

| Option | Parallel requests | Upper bound if speed held |
| --- | --- | --- |
| Pay-as-you-go key | 4 (and 30 requests per minute) | 4 streams × about 100 tokens per second each |
| Hourly key (Gemma 4 VL 26B) | 1 stream, 300,000 tokens per minute | not published; measure it |
| Flat plan session | 1 request at a time | about 100 tokens per second |

To scale a flat plan, you add sessions, one per agent working in parallel; see [unlimited LLM API sessions](https://broadrouter.com/guides/unlimited-llm-api-sessions/) and [running parallel coding agents](https://broadrouter.com/blog/run-parallel-coding-agents-flat-rate/). That speed is BroadNet’s stated typical figure for DeepSeek V4.1 Flash, not a guarantee for other models or under load.

## Questions

### Is throughput the same as tokens per second?

Only at system level. Per-stream tokens per second is the speed one request sees. Throughput adds up all streams. NVIDIA calls these TPS per user and total TPS per system.

### How do I get more throughput from an LLM API?

Run more requests in parallel, up to your concurrency and rate limits, and keep each request short. If each request already uses its slot fully, more slots, keys or sessions are the only way up.

### Does higher throughput mean lower latency?

No. Throughput is total work per second across all requests; latency is how long one request takes. Serving more requests at once raises total throughput but can slow each request, which is why NVIDIA reports TPS per user and total TPS separately.

## See also

-   [Tokens per second](https://broadrouter.com/glossary/tokens-per-second/) (term)
-   [Concurrency](https://broadrouter.com/glossary/concurrency/) (term)
-   [Latency](https://broadrouter.com/glossary/latency/) (term)
-   [KV cache](https://broadrouter.com/glossary/kv-cache/) (term)
-   [Batch API](https://broadrouter.com/glossary/batch-api/) (term)
-   [Speculative decoding](https://broadrouter.com/glossary/speculative-decoding/) (term)
-   [How the unlimited LLM API plans work for agents](https://broadrouter.com/guides/unlimited-llm-api-sessions/) (guide)
-   [BroadNet Inference error codes and when to retry](https://broadrouter.com/guides/errors-and-retries/) (guide)
-   [How to run parallel coding agents on a flat-rate LLM API](https://broadrouter.com/blog/run-parallel-coding-agents-flat-rate/) (article)
-   [LLM latency: TTFT vs tokens per second vs end-to-end time](https://broadrouter.com/blog/llm-latency-ttft-throughput/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-throughput&utm_content=page-cta) ·

**Sources**

-   [NVIDIA NIM, LLM benchmarking metrics (total TPS per system, TPS per user, RPS)](https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html)
-   [Kwon et al., PagedAttention and vLLM, arXiv 2309.06180](https://arxiv.org/abs/2309.06180)
-   [BroadNet Inference API reference, limits](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-throughput&utm_content=sources#limits)
-   [BroadNet Inference plans (typical speed, one request per session)](https://inference.broadnet.ai/plans?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-throughput&utm_content=sources)
