---
title: "What is concurrency in an LLM API? In-flight request limits"
description: "Concurrency is how many requests run at the same moment. Why LLM APIs cap it, how it relates to rate limits and latency, and how to size workers and agents."
url: https://broadrouter.com/glossary/concurrency/
lang: en
updated: 2026-11-14
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Concurrency

Updated 2026-11-14

Concurrency is the number of requests a client has in flight at the same time. LLM APIs often cap it per key or plan, separately from requests per minute, because each open request holds GPU capacity until its response has finished.

Also called: concurrent requests, parallel requests, in-flight requests, concurrency limit

Rate limits count requests over time. Concurrency counts requests at one instant. LLM requests can take many seconds, so the second limit often bites first.

## How it works

Little’s law links the three numbers:

`requests in flight = request rate × average time per request`

Take a hypothetical key that allows 5 requests in flight and 60 requests a minute, where each request takes 20 seconds. The concurrency cap allows at most 5 ÷ 20 = 0.25 requests a second, or 15 a minute. You never reach 60 a minute, because concurrency is the binding limit. With 2-second requests, the same cap allows 150 a minute, and the rate limit binds instead. The [rate limit calculator](https://broadrouter.com/tools/llm-rate-limit-calculator/) runs this sum for your own numbers.

Long outputs, [reasoning tokens](https://broadrouter.com/glossary/reasoning-tokens/) and long prompts all make requests take longer, so they use up concurrency faster. A [streamed](https://broadrouter.com/glossary/streaming/) reply holds its slot until it ends.

## Why it matters

Providers cap concurrency because GPU memory and compute are shared. NVIDIA’s benchmarks show that as concurrency on a server rises, total output goes up while each request’s [tokens per second](https://broadrouter.com/glossary/tokens-per-second/) go down. A per-key cap keeps one client from slowing everyone.

For your app, concurrency sets the size of worker pools and how many agents can run side by side. Size your pool to the cap, not above it, or extra workers just collect 429 errors.

## On BroadNet Inference

| Access | Concurrency |
| --- | --- |
| Pay-as-you-go key | 4 requests in flight, plus 30 requests a minute |
| Hourly key | One stream |
| Unlimited plan | One request at a time per session |

The plans sell concurrency directly. A session has no token meter and runs one request at a time, the way one agent works. Requests in the same session queue behind each other. To run agents in parallel, buy one session per agent: an orchestrator plus three sub-agents needs four sessions. One key can hold several sessions.

Read [how unlimited sessions work](https://broadrouter.com/guides/unlimited-llm-api-sessions/) and [how to run parallel coding agents on a flat rate](https://broadrouter.com/blog/run-parallel-coding-agents-flat-rate/). Agents spend much of their time running tools, so measure real overlap before buying more sessions.

## Common mistakes

-   Starting one worker per task with no pool. A batch of 100 tasks becomes 100 requests at once.
-   Setting client timeouts shorter than a long reasoning reply. The client gives up while the slot is still busy.

## Questions

### What happens when I exceed the concurrency limit?

Depending on the provider, the extra request is refused, often with a 429, or it waits in a queue until a slot frees up.

### Does streaming use a concurrency slot?

Yes. A streamed response holds its slot until the last chunk arrives, so long outputs keep a slot busy longer.

### How many concurrent requests does an AI coding agent need?

One agent usually needs one, because it waits for each reply before its next step. An orchestrator with three sub-agents running at once needs up to four. It needs four if the orchestrator sends requests while they run, and three if it waits for them.

## See also

-   [Rate limit](https://broadrouter.com/glossary/rate-limit/) (term)
-   [Tokens per second](https://broadrouter.com/glossary/tokens-per-second/) (term)
-   [Streaming](https://broadrouter.com/glossary/streaming/) (term)
-   [Flat-rate LLM plan](https://broadrouter.com/glossary/flat-rate-llm-plan/) (term)
-   [Throughput](https://broadrouter.com/glossary/throughput/) (term)
-   [Latency](https://broadrouter.com/glossary/latency/) (term)
-   [AI agent](https://broadrouter.com/glossary/agent/) (term)
-   [How the unlimited LLM API plans work for agents](https://broadrouter.com/guides/unlimited-llm-api-sessions/) (guide)
-   [BroadNet Inference error codes and when to retry](https://broadrouter.com/guides/errors-and-retries/) (guide)
-   [How to run parallel coding agents on a flat-rate LLM API](https://broadrouter.com/blog/run-parallel-coding-agents-flat-rate/) (article)
-   [What "unlimited" really means in an LLM API plan](https://broadrouter.com/blog/what-unlimited-llm-api-really-means/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-concurrency&utm_content=page-cta) ·

**Sources**

-   [Little's law (Wikipedia)](https://en.wikipedia.org/wiki/Little%27s_law)
-   [NVIDIA NIM, LLM benchmarking metrics (concurrency and throughput)](https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html)
-   [BroadNet Inference API reference, limits](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-concurrency&utm_content=sources#limits)
-   [BroadNet Inference plans (sessions)](https://inference.broadnet.ai/plans?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-concurrency&utm_content=sources)
