A 429 means your traffic crossed a rate limit: requests per minute, requests in flight, tokens per minute or spend. Fix it in two halves. Retry well: capped exponential backoff with random jitter, honouring Retry-After. Send less at once: a semaphore at the key’s in-flight limit, plus a queue. Both are below, with code.
For a full status-code reference, see the errors and retries guide; for multi-hour agent runs with checkpoints and timeouts, see long-running agents: timeouts and retries.
What is the difference between 429 and 5xx?
| Status | Whose problem | Retry? | How |
|---|---|---|---|
| 429 | Your traffic crossed a limit | Yes | Backoff with jitter; honour Retry-After; lower concurrency |
| 500 / 502 / 503 / 504 | Server side | Yes | Bounded exponential backoff |
| Connection reset, timeout | Network or server | Usually | Backoff; for streams, only if a new bill is acceptable |
| 400 / 422 | Bad request | No | Fix the payload |
| 401 / 402 / 403 | Key, balance or policy | No | Fix the key, fund the account, fix the allow-list |
| 413 | Request too large | No | Shrink it |
On BroadNet Inference a 429 can mean any of four limits: rate, concurrency, token rate or spend. A spend limit will not clear in 30 seconds. BroadNet’s docs do not say which field of the error body names the limit, so log the body and compare it with the limits shown in your key details.
Our open dataset of LLM API free tiers and rate limits compares what 14 providers publish about free access, starting rate limits, card requirements and training on free-tier data, with the source and date for each value.
What limits cause 429s?
Know the numbers you are working against. On BroadNet Inference:
| Key type | Requests / minute | In flight | Other |
|---|---|---|---|
| Pay-as-you-go | 30 | 4 | Shared spend budget |
| Hourly key | 120 | 1 stream | 300,000 tokens / minute |
| Unlimited plan session | — | 1 request per session | No token meter |
The in-flight number matters most for LLMs. A streamed answer holds its slot until the last token arrives, so a long answer keeps it busy for as long as the model takes to write it (BroadNet gives a typical speed of about 100 tokens per second). Send more parallel calls than the key’s 4 slots and the extra ones get 429, even though you sent only a handful of requests that minute. That is a concurrency problem, and backoff alone handles it badly. To size workers against a limit before you launch, use the free LLM rate limit calculator. For a fixed job of many items, the LLM batch job planner turns the same limits into a run time. Other providers count differently (tokens a minute, requests a day or requests in flight); LLM API rate limits by provider compares 13 of them from their own docs.
How do you write backoff with jitter?
Plain exponential backoff waits 1, 2, 4, 8 seconds. If a hundred workers were throttled at the same moment, they all come back at the same moment. AWS’s simulation of 100 contending clients found that full jitter cut the number of calls by more than half and finished sooner than unjittered backoff. The simplest version, full jitter, waits a random time between zero and the exponential cap:
import os, random, time
from openai import OpenAI, RateLimitError, APIStatusError, APIConnectionError
client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"], max_retries=0)
RETRYABLE = {429, 500, 502, 503, 504}
def call_with_backoff(max_attempts=6, base=1.0, cap=60.0, **kwargs):
for attempt in range(max_attempts):
try:
return client.chat.completions.create(**kwargs)
except (RateLimitError, APIStatusError, APIConnectionError) as e:
status = getattr(e, "status_code", None)
if status is not None and status not in RETRYABLE:
raise # 400, 401, 403...: retrying will not help
if attempt == max_attempts - 1:
raise
retry_after = None
if getattr(e, "response", None) is not None:
retry_after = e.response.headers.get("retry-after")
wait = random.uniform(0, min(cap, base * 2 ** attempt))
if retry_after and retry_after.isdigit():
wait = max(wait, float(retry_after))
time.sleep(wait)
max_retries=0 turns off the SDK’s own retries so you do not retry twice. If you are happy with the SDK’s built-in behaviour, the opposite works too: leave your loop out and set max_retries=4 on the client, as the Python SDK guide shows. The official Python and Node SDKs retry connection errors, 408, 409, 429 and 5xx twice by default.
Retry-After is a standard header (RFC 9110) that may hold seconds or a date. BroadNet’s docs do not promise it, so the code treats it as optional.
How do you stop 429s before they happen?
Cap requests in flight at the key’s limit. In Python, an asyncio.Semaphore does it with one object and one async with block. The code uses 4 as an example; set it to your key’s in-flight limit (4 on a BroadNet pay-as-you-go key):
import asyncio, os
from openai import AsyncOpenAI
client = AsyncOpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"], max_retries=4)
slots = asyncio.Semaphore(4) # match the key's in-flight limit
async def ask(prompt):
async with slots:
r = await client.chat.completions.create(
model="broadnet/gemma-4-26b-a4b",
messages=[{"role": "user", "content": prompt}],
)
return r.choices[0].message.content
async def main(prompts):
return await asyncio.gather(*(ask(p) for p in prompts))
results = asyncio.run(main(["Summarise A", "Summarise B", "Summarise C"]))
gather starts every task, but only as many run at once as the semaphore allows. The rest wait in the semaphore’s queue. In Node, the p-limit package does the same, again with 4 as the example value:
// ES module (top-level await). npm install openai p-limit
import OpenAI from "openai";
import pLimit from "p-limit";
const client = new OpenAI({ baseURL: "https://inference.broadnet.ai/v1", apiKey: process.env.BROADNET_API_KEY, maxRetries: 4 });
const limit = pLimit(4); // match the key's in-flight limit
async function ask(prompt) {
const r = await client.chat.completions.create({
model: "broadnet/gemma-4-26b-a4b",
messages: [{ role: "user", content: prompt }],
});
return r.choices[0].message.content;
}
const prompts = ["Summarise A", "Summarise B", "Summarise C"];
const results = await Promise.all(prompts.map((p) => limit(() => ask(p))));
For a per-minute cap as well, add a token bucket that releases one permit every 60 / rpm seconds, where rpm is your key’s requests-a-minute limit (30 on a BroadNet pay-as-you-go key).
When do you need a real queue?
A semaphore lives in one process. Once several workers or servers share one key, the limit is shared and each process only sees its own traffic. Then:
- Put jobs on a shared queue (Redis, SQS, a database table).
- Run a fixed number of consumers equal to the key’s in-flight limit.
- Give each job an ID and store results by ID, so a retried job does not run twice.
- Send interactive traffic and batch traffic through separate keys or separate queues, so a batch backlog never blocks a user waiting on a reply.
For a bulk job of thousands of items with no Batch API, see the resumable batch runner.
Which retry mistakes turn 429s into outages?
- Unbounded retries. A retry loop with no cap during a long outage multiplies your load and, on per-token billing, your bill.
- Retrying a half-finished stream blindly. If the stream already returned tokens, a retry is a fresh inference and the user may see text twice. See the streaming post for detecting incomplete streams.
- Retrying account errors. A 401 for an exhausted pay-as-you-go allowance or a 402 for insufficient balance will not fix itself.
Log every retry with its status, wait time and the X-Request-ID response header, which BroadNet’s docs ask you to keep for support. A rising retry rate is the earliest warning that you need more capacity; usage logging shows how to record it.
Where BroadNet Inference fits
BroadNet’s limits are published and small enough to design against: 4 requests in flight on a pay-as-you-go key. That suits apps and agents, and it is too tight for a bulk job that wants hundreds of parallel calls on one key. For agents, an unlimited plan has no token meter and runs one request at a time per session; you add sessions for parallel work, as described in running parallel coding agents. Current limits are in the API reference.
