Rate limits count requests over time. Concurrency counts requests at one instant. LLM requests can take many seconds, so the second limit often bites first.
How it works
Little’s law links the three numbers:
requests in flight = request rate × average time per request
Take a hypothetical key that allows 5 requests in flight and 60 requests a minute, where each request takes 20 seconds. The concurrency cap allows at most 5 ÷ 20 = 0.25 requests a second, or 15 a minute. You never reach 60 a minute, because concurrency is the binding limit. With 2-second requests, the same cap allows 150 a minute, and the rate limit binds instead. The rate limit calculator runs this sum for your own numbers.
Long outputs, reasoning tokens and long prompts all make requests take longer, so they use up concurrency faster. A streamed reply holds its slot until it ends.
Why it matters
Providers cap concurrency because GPU memory and compute are shared. NVIDIA’s benchmarks show that as concurrency on a server rises, total output goes up while each request’s tokens per second go down. A per-key cap keeps one client from slowing everyone.
For your app, concurrency sets the size of worker pools and how many agents can run side by side. Size your pool to the cap, not above it, or extra workers just collect 429 errors.
On BroadNet Inference
| Access | Concurrency |
|---|---|
| Pay-as-you-go key | 4 requests in flight, plus 30 requests a minute |
| Hourly key | One stream |
| Unlimited plan | One request at a time per session |
The plans sell concurrency directly. A session has no token meter and runs one request at a time, the way one agent works. Requests in the same session queue behind each other. To run agents in parallel, buy one session per agent: an orchestrator plus three sub-agents needs four sessions. One key can hold several sessions.
Read how unlimited sessions work and how to run parallel coding agents on a flat rate. Agents spend much of their time running tools, so measure real overlap before buying more sessions.
Common mistakes
- Starting one worker per task with no pool. A batch of 100 tasks becomes 100 requests at once.
- Setting client timeouts shorter than a long reasoning reply. The client gives up while the slot is still busy.