BroadRouter
Get started

Concurrency

Updated

Concurrency is the number of requests a client has in flight at the same time. LLM APIs often cap it per key or plan, separately from requests per minute, because each open request holds GPU capacity until its response has finished.

Also called: concurrent requests, parallel requests, in-flight requests, concurrency limit

Rate limits count requests over time. Concurrency counts requests at one instant. LLM requests can take many seconds, so the second limit often bites first.

How it works

Little’s law links the three numbers:

requests in flight = request rate × average time per request

Take a hypothetical key that allows 5 requests in flight and 60 requests a minute, where each request takes 20 seconds. The concurrency cap allows at most 5 ÷ 20 = 0.25 requests a second, or 15 a minute. You never reach 60 a minute, because concurrency is the binding limit. With 2-second requests, the same cap allows 150 a minute, and the rate limit binds instead. The rate limit calculator runs this sum for your own numbers.

Long outputs, reasoning tokens and long prompts all make requests take longer, so they use up concurrency faster. A streamed reply holds its slot until it ends.

Why it matters

Providers cap concurrency because GPU memory and compute are shared. NVIDIA’s benchmarks show that as concurrency on a server rises, total output goes up while each request’s tokens per second go down. A per-key cap keeps one client from slowing everyone.

For your app, concurrency sets the size of worker pools and how many agents can run side by side. Size your pool to the cap, not above it, or extra workers just collect 429 errors.

On BroadNet Inference

Access Concurrency
Pay-as-you-go key 4 requests in flight, plus 30 requests a minute
Hourly key One stream
Unlimited plan One request at a time per session

The plans sell concurrency directly. A session has no token meter and runs one request at a time, the way one agent works. Requests in the same session queue behind each other. To run agents in parallel, buy one session per agent: an orchestrator plus three sub-agents needs four sessions. One key can hold several sessions.

Read how unlimited sessions work and how to run parallel coding agents on a flat rate. Agents spend much of their time running tools, so measure real overlap before buying more sessions.

Common mistakes

  • Starting one worker per task with no pool. A batch of 100 tasks becomes 100 requests at once.
  • Setting client timeouts shorter than a long reasoning reply. The client gives up while the slot is still busy.

Questions

What happens when I exceed the concurrency limit?

Depending on the provider, the extra request is refused, often with a 429, or it waits in a queue until a slot frees up.

Does streaming use a concurrency slot?

Yes. A streamed response holds its slot until the last chunk arrives, so long outputs keep a slot busy longer.

How many concurrent requests does an AI coding agent need?

One agent usually needs one, because it waits for each reply before its next step. An orchestrator with three sub-agents running at once needs up to four. It needs four if the orchestrator sends requests while they run, and three if it waits for them.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity