When an API says “slow down”, retrying at once makes things worse. Exponential backoff waits longer each time.
How it works
After a failure, wait a base delay, then double it on each new failure, up to a maximum. With a 1-second base and a 32-second cap, the waits run 1, 2, 4, 8, 16, 32, 32 seconds. Stop after a fixed number of attempts and surface the error.
Backoff alone is not enough. If 100 clients fail at the same moment, they all retry at the same moments too. Marc Brooker’s AWS post compares “Full Jitter” (a random wait between zero and the current cap), “Equal Jitter” and “Decorrelated Jitter”. Equal Jitter came out weakest, and Full Jitter “uses less work, but slightly more time.” His conclusion: jittered backoff “should be considered a standard approach for remote clients.”
import random, time
def call_with_backoff(fn, attempts=6, base=1.0, cap=32.0):
for n in range(attempts):
try:
return fn()
except RetryableError:
if n == attempts - 1:
raise
time.sleep(random.uniform(0, min(cap, base * 2 ** n))) # full jitter
Why it matters
- 429s count too. OpenAI’s rate-limit guide warns that “unsuccessful requests contribute to your per-minute limit, so continuously resending a request won’t work.” Hammering a rate limit keeps you locked out.
- Retries multiply. AWS’s Builders’ Library notes that retries are “selfish” and recommends retrying at a single point in the stack; retrying at every layer can multiply load many times over.
- Side effects. Retrying a call that charges money is safe only with an idempotency key.
- Prevention beats retry. Cap requests in flight below your concurrency limit so 429s are rare in the first place.
On BroadNet Inference
BroadNet’s error table says to answer a 429 with “response details and bounded backoff” and to retry 500, 502, 503 and 504 “with bounded exponential backoff.” A pay-as-you-go key allows 30 requests per minute and 4 in flight. For 400, 401, 403, 413 and 422, fix the cause first. Keep the X-Request-ID header for support. The errors and retries guide sets SDK retry counts, and how to handle LLM API 429s has limiter and queue code. The rate limit calculator shows how much traffic fits under a limit.