---
title: "Handle LLM API 429 rate limits - backoff, jitter, queues"
description: "429 vs 5xx, exponential backoff with full jitter, concurrency limiters and queues for LLM APIs, with Python and Node code you can copy into production."
url: https://broadrouter.com/blog/handle-llm-api-429-rate-limits/
lang: en
updated: 2026-11-14
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# How to handle LLM API 429 errors: backoff, jitter and limiters

By the BroadNet Inference team · 5 min read · Published 2026-11-14 · Updated 2026-11-15

A 429 means you hit a rate, concurrency or spend limit. Retry it with capped exponential backoff and random jitter, and honour Retry-After when sent. Then prevent most 429s: cap requests in flight with a semaphore at the key's limit and queue the rest. Retry 5xx the same way; fix other 4xx errors instead.

## Key takeaways

-   Retry 429 and 5xx with bounded exponential backoff. Do not retry 400, 401, 403, 413 or 422: fix the request or account.
-   Full jitter (a random wait between zero and the exponential cap) stops many clients retrying at the same instant.
-   Most 429s come from concurrency, not volume. A semaphore set to the key's in-flight limit removes them at the source.
-   A pay-as-you-go BroadNet key allows 30 requests a minute and 4 in flight; plan sessions run one request at a time.
-   Retries are new inferences. On per-token billing each one is billed again, so cap attempts.

**On this page**

1.  What is the difference between 429 and 5xx?
2.  What limits cause 429s?
3.  How do you write backoff with jitter?
4.  How do you stop 429s before they happen?
5.  When do you need a real queue?
6.  Which retry mistakes turn 429s into outages?
7.  Where BroadNet Inference fits

A 429 means your traffic crossed a [rate limit](https://broadrouter.com/glossary/rate-limit/): requests per minute, requests in flight, tokens per minute or spend. Fix it in two halves. **Retry well**: capped exponential backoff with random jitter, honouring `Retry-After`. **Send less at once**: a semaphore at the key’s in-flight limit, plus a queue. Both are below, with code.

For a full status-code reference, see the [errors and retries guide](https://broadrouter.com/guides/errors-and-retries/); for multi-hour agent runs with checkpoints and timeouts, see [long-running agents: timeouts and retries](https://broadrouter.com/blog/long-running-agents-timeouts-retries/).

## What is the difference between 429 and 5xx?

| Status | Whose problem | Retry? | How |
| --- | --- | --- | --- |
| 429 | Your traffic crossed a limit | Yes | Backoff with jitter; honour `Retry-After`; lower concurrency |
| 500 / 502 / 503 / 504 | Server side | Yes | Bounded exponential backoff |
| Connection reset, timeout | Network or server | Usually | Backoff; for streams, only if a new bill is acceptable |
| 400 / 422 | Bad request | No | Fix the payload |
| 401 / 402 / 403 | Key, balance or policy | No | Fix the key, fund the account, fix the allow-list |
| 413 | Request too large | No | Shrink it |

On BroadNet Inference a 429 can mean any of four limits: rate, concurrency, token rate or spend. A spend limit will not clear in 30 seconds. BroadNet’s docs do not say which field of the error body names the limit, so log the body and compare it with the limits shown in your key details.

Our open dataset of [LLM API free tiers and rate limits](https://broadrouter.com/data/llm-api-free-tiers-and-rate-limits/) compares what 14 providers publish about free access, starting rate limits, card requirements and training on free-tier data, with the source and date for each value.

## What limits cause 429s?

Know the numbers you are working against. On BroadNet Inference:

| Key type | Requests / minute | In flight | Other |
| --- | --- | --- | --- |
| Pay-as-you-go | 30 | 4 | Shared spend budget |
| Hourly key | 120 | 1 stream | 300,000 tokens / minute |
| Unlimited plan session | — | 1 request per session | No token meter |

The in-flight number matters most for LLMs. A streamed answer holds its slot until the last token arrives, so a long answer keeps it busy for as long as the model takes to write it (BroadNet gives a typical speed of about 100 tokens per second). Send more parallel calls than the key’s 4 slots and the extra ones get 429, even though you sent only a handful of requests that minute. That is a [concurrency](https://broadrouter.com/glossary/concurrency/) problem, and backoff alone handles it badly. To size workers against a limit before you launch, use the free [LLM rate limit calculator](https://broadrouter.com/tools/llm-rate-limit-calculator/). For a fixed job of many items, the [LLM batch job planner](https://broadrouter.com/tools/llm-batch-job-planner/) turns the same limits into a run time. Other providers count differently (tokens a minute, requests a day or requests in flight); [LLM API rate limits by provider](https://broadrouter.com/blog/llm-api-rate-limits-by-provider/) compares 13 of them from their own docs.

## How do you write backoff with jitter?

Plain [exponential backoff](https://broadrouter.com/glossary/exponential-backoff/) waits 1, 2, 4, 8 seconds. If a hundred workers were throttled at the same moment, they all come back at the same moment. AWS’s simulation of 100 contending clients found that full jitter cut the number of calls by more than half and finished sooner than unjittered backoff. The simplest version, **full jitter**, waits a random time between zero and the exponential cap:

```python
import os, random, time
from openai import OpenAI, RateLimitError, APIStatusError, APIConnectionError

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"], max_retries=0)

RETRYABLE = {429, 500, 502, 503, 504}

def call_with_backoff(max_attempts=6, base=1.0, cap=60.0, **kwargs):
    for attempt in range(max_attempts):
        try:
            return client.chat.completions.create(**kwargs)
        except (RateLimitError, APIStatusError, APIConnectionError) as e:
            status = getattr(e, "status_code", None)
            if status is not None and status not in RETRYABLE:
                raise  # 400, 401, 403...: retrying will not help
            if attempt == max_attempts - 1:
                raise
            retry_after = None
            if getattr(e, "response", None) is not None:
                retry_after = e.response.headers.get("retry-after")
            wait = random.uniform(0, min(cap, base * 2 ** attempt))
            if retry_after and retry_after.isdigit():
                wait = max(wait, float(retry_after))
            time.sleep(wait)
```

`max_retries=0` turns off the SDK’s own retries so you do not retry twice. If you are happy with the SDK’s built-in behaviour, the opposite works too: leave your loop out and set `max_retries=4` on the client, as the [Python SDK guide](https://broadrouter.com/guides/openai-python-sdk/) shows. The official Python and Node SDKs retry connection errors, 408, 409, 429 and 5xx twice by default.

`Retry-After` is a standard header (RFC 9110) that may hold seconds or a date. BroadNet’s docs do not promise it, so the code treats it as optional.

## How do you stop 429s before they happen?

Cap requests in flight at the key’s limit. In Python, an `asyncio.Semaphore` does it with one object and one `async with` block. The code uses 4 as an example; set it to your key’s in-flight limit (4 on a BroadNet pay-as-you-go key):

```python
import asyncio, os
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"], max_retries=4)
slots = asyncio.Semaphore(4)  # match the key's in-flight limit

async def ask(prompt):
    async with slots:
        r = await client.chat.completions.create(
            model="broadnet/gemma-4-26b-a4b",
            messages=[{"role": "user", "content": prompt}],
        )
        return r.choices[0].message.content

async def main(prompts):
    return await asyncio.gather(*(ask(p) for p in prompts))

results = asyncio.run(main(["Summarise A", "Summarise B", "Summarise C"]))
```

`gather` starts every task, but only as many run at once as the semaphore allows. The rest wait in the semaphore’s queue. In Node, the `p-limit` package does the same, again with 4 as the example value:

```javascript
// ES module (top-level await). npm install openai p-limit
import OpenAI from "openai";
import pLimit from "p-limit";

const client = new OpenAI({ baseURL: "https://inference.broadnet.ai/v1", apiKey: process.env.BROADNET_API_KEY, maxRetries: 4 });
const limit = pLimit(4); // match the key's in-flight limit

async function ask(prompt) {
  const r = await client.chat.completions.create({
    model: "broadnet/gemma-4-26b-a4b",
    messages: [{ role: "user", content: prompt }],
  });
  return r.choices[0].message.content;
}

const prompts = ["Summarise A", "Summarise B", "Summarise C"];
const results = await Promise.all(prompts.map((p) => limit(() => ask(p))));
```

For a per-minute cap as well, add a token bucket that releases one permit every 60 / rpm seconds, where rpm is your key’s requests-a-minute limit (30 on a BroadNet pay-as-you-go key).

## When do you need a real queue?

A semaphore lives in one process. Once several workers or servers share one key, the limit is shared and each process only sees its own traffic. Then:

1.  Put jobs on a shared queue (Redis, SQS, a database table).
2.  Run a fixed number of consumers equal to the key’s in-flight limit.
3.  Give each job an ID and store results by ID, so a retried job does not run twice.
4.  Send interactive traffic and batch traffic through separate keys or separate queues, so a batch backlog never blocks a user waiting on a reply.

For a bulk job of thousands of items with no Batch API, see [the resumable batch runner](https://broadrouter.com/blog/llm-batch-processing-without-batch-api/).

## Which retry mistakes turn 429s into outages?

-   **Unbounded retries.** A retry loop with no cap during a long outage multiplies your load and, on per-token billing, your bill.
-   **Retrying a half-finished stream blindly.** If the stream already returned tokens, a retry is a fresh inference and the user may see text twice. See [the streaming post](https://broadrouter.com/blog/llm-streaming-sse-explained/) for detecting incomplete streams.
-   **Retrying account errors.** A 401 for an exhausted pay-as-you-go allowance or a 402 for insufficient balance will not fix itself.

Log every retry with its status, wait time and the `X-Request-ID` response header, which BroadNet’s docs ask you to keep for support. A rising retry rate is the earliest warning that you need more capacity; [usage logging](https://broadrouter.com/blog/llm-api-usage-logging/) shows how to record it.

## Where BroadNet Inference fits

BroadNet’s limits are published and small enough to design against: 4 requests in flight on a pay-as-you-go key. That suits apps and agents, and it is too tight for a bulk job that wants hundreds of parallel calls on one key. For agents, an [unlimited plan](https://broadrouter.com/guides/unlimited-llm-api-sessions/) has no token meter and runs one request at a time per session; you add sessions for parallel work, as described in [running parallel coding agents](https://broadrouter.com/blog/run-parallel-coding-agents-flat-rate/). Current limits are in the [API reference](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-handle-llm-api-429-rate-limits&utm_content=body#limits).

Free tools for this

-   [OpenAI to BroadNet migration helper](https://broadrouter.com/tools/openai-to-broadnet-migration/) Paste OpenAI SDK code; get base URL, key and model edits, a diff and the checks that remain.
-   [max\_tokens calculator for reasoning models](https://broadrouter.com/tools/reasoning-token-budget/) Answer length + reasoning budget + margin = a max\_tokens that does not cut the answer off.
-   [JSON schema generator for LLM structured output](https://broadrouter.com/tools/json-schema-generator/) Sample JSON to strict schema, Pydantic, Zod and a ready request.
-   [LLM JSON output validator](https://broadrouter.com/tools/llm-json-output-validator/) Parse errors by line and column, safe repairs, cut-off detection and JSON Schema checks for model output.

[All free tools](https://broadrouter.com/tools/)

## Terms in this article

[Rate limit](https://broadrouter.com/glossary/rate-limit/)

A rate limit is a cap an API places on how much a client can use it in a time window, such as requests per minute or tokens per minute. Requests over the limit are refused, usually with HTTP 429 Too Many Requests, until usage falls back under it.

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[Pay-as-you-go](https://broadrouter.com/glossary/pay-as-you-go/)

Pay-as-you-go is a pricing model in which you pay only for what you use, metered per unit such as tokens, requests or audio minutes, with no subscription or minimum commitment. Most LLM APIs bill this way, from a prepaid balance or a periodic invoice.

[Concurrency](https://broadrouter.com/glossary/concurrency/)

Concurrency is the number of requests a client has in flight at the same time. LLM APIs often cap it per key or plan, separately from requests per minute, because each open request holds GPU capacity until its response has finished.

[Exponential backoff](https://broadrouter.com/glossary/exponential-backoff/)

Exponential backoff is a retry strategy in which a client waits longer after each failed attempt, typically doubling the delay up to a cap, before trying again. Adding random jitter to each wait spreads out retries from many clients so they do not hit the server at the same moment.

[Tool calling](https://broadrouter.com/glossary/tool-calling/)

Tool calling, also called function calling, is an LLM API feature in which the model returns a structured request to run a function the developer has defined, with JSON arguments. The application runs the function and sends the result back; the model never runs code itself.

[AI coding agent](https://broadrouter.com/glossary/coding-agent/)

An AI coding agent is a software tool in which a language model works on a codebase in a loop: it reads files, edits them, runs commands such as tests, reads the output and decides the next step until the task is done or a limit is reached.

[Batch API](https://broadrouter.com/glossary/batch-api/)

A batch API is an asynchronous LLM endpoint that accepts many requests at once, usually as a file, processes them within a set window instead of immediately, and returns the results together. Providers such as OpenAI and Anthropic price batch requests at a discount to their standard APIs.

## Questions

### What does HTTP 429 mean on an LLM API?

Too Many Requests. You crossed a limit on requests per minute, requests in flight, tokens per minute or spend. The request was not processed, so it is safe to send again after a wait.

### How long should I wait before retrying a 429?

Use Retry-After if the server sends it. Otherwise start near one second, double each attempt, cap around 30 to 60 seconds, and pick a random wait between zero and that cap. Stop after four to six attempts.

### Should I retry 500 and 503 errors from an LLM API?

Yes, with the same bounded backoff. They are server-side. A 400, 401, 403 or 422 will fail the same way every time, so fix it instead.

### Does the OpenAI SDK retry 429 automatically?

Yes. The official Python and Node SDKs retry connection errors, 408, 409, 429 and 5xx twice by default with a short exponential backoff. Raise max\_retries for batch jobs and add your own concurrency limit.

### Why do I get 429 when I am under the requests-per-minute limit?

Usually the concurrency limit. Long streaming responses stay in flight for many seconds, so a handful of parallel calls can hit the in-flight cap while the per-minute count looks low.

## Keep reading

-   [Long-running agents: timeouts, retries and checkpoints that hold](https://broadrouter.com/blog/long-running-agents-timeouts-retries/)
-   [LLM streaming explained: SSE chunks, deltas, usage and cancel](https://broadrouter.com/blog/llm-streaming-sse-explained/)
-   [How to run parallel coding agents on a flat-rate LLM API](https://broadrouter.com/blog/run-parallel-coding-agents-flat-rate/)
-   [LLM API usage logging: cost per user and feature, PII redacted](https://broadrouter.com/blog/llm-api-usage-logging/)
-   [LLM batch processing without a Batch API: a resumable runner](https://broadrouter.com/blog/llm-batch-processing-without-batch-api/)
-   [A private endpoint starts with a clear workload](https://broadrouter.com/blog/private-endpoint-planning/)
-   [Before fine-tuning: define the problem to solve](https://broadrouter.com/blog/fine-tuning-readiness/)
-   [Guide: BroadNet Inference error codes and when to retry](https://broadrouter.com/guides/errors-and-retries/)
-   [Guide: Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/)
-   [Guide: How the unlimited LLM API plans work for agents](https://broadrouter.com/guides/unlimited-llm-api-sessions/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-handle-llm-api-429-rate-limits&utm_content=page-cta) ·

**Sources**

-   [BroadNet Inference API reference, errors and limits (fetched 2026-11-14)](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-handle-llm-api-429-rate-limits&utm_content=sources#errors)
-   [AWS Architecture Blog, Exponential Backoff and Jitter (fetched 2026-11-14)](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/)
-   [RFC 6585, status code 429 Too Many Requests (fetched 2026-11-14)](https://www.rfc-editor.org/rfc/rfc6585#section-4)
-   [RFC 9110, Retry-After header (fetched 2026-11-14)](https://www.rfc-editor.org/rfc/rfc9110#field.retry-after)
-   [openai-python README, automatic retries (fetched 2026-11-14)](https://github.com/openai/openai-python)
-   [openai-node README, retries (fetched 2026-11-14)](https://github.com/openai/openai-node)
