BroadRouter
Get started

Exponential backoff

Updated

Exponential backoff is a retry strategy in which a client waits longer after each failed attempt, typically doubling the delay up to a cap, before trying again. Adding random jitter to each wait spreads out retries from many clients so they do not hit the server at the same moment.

Also called: backoff and jitter, retry with backoff, jittered backoff, capped exponential backoff, full jitter

When an API says “slow down”, retrying at once makes things worse. Exponential backoff waits longer each time.

How it works

After a failure, wait a base delay, then double it on each new failure, up to a maximum. With a 1-second base and a 32-second cap, the waits run 1, 2, 4, 8, 16, 32, 32 seconds. Stop after a fixed number of attempts and surface the error.

Backoff alone is not enough. If 100 clients fail at the same moment, they all retry at the same moments too. Marc Brooker’s AWS post compares “Full Jitter” (a random wait between zero and the current cap), “Equal Jitter” and “Decorrelated Jitter”. Equal Jitter came out weakest, and Full Jitter “uses less work, but slightly more time.” His conclusion: jittered backoff “should be considered a standard approach for remote clients.”

import random, time

def call_with_backoff(fn, attempts=6, base=1.0, cap=32.0):
    for n in range(attempts):
        try:
            return fn()
        except RetryableError:
            if n == attempts - 1:
                raise
            time.sleep(random.uniform(0, min(cap, base * 2 ** n)))  # full jitter

Why it matters

  • 429s count too. OpenAI’s rate-limit guide warns that “unsuccessful requests contribute to your per-minute limit, so continuously resending a request won’t work.” Hammering a rate limit keeps you locked out.
  • Retries multiply. AWS’s Builders’ Library notes that retries are “selfish” and recommends retrying at a single point in the stack; retrying at every layer can multiply load many times over.
  • Side effects. Retrying a call that charges money is safe only with an idempotency key.
  • Prevention beats retry. Cap requests in flight below your concurrency limit so 429s are rare in the first place.

On BroadNet Inference

BroadNet’s error table says to answer a 429 with “response details and bounded backoff” and to retry 500, 502, 503 and 504 “with bounded exponential backoff.” A pay-as-you-go key allows 30 requests per minute and 4 in flight. For 400, 401, 403, 413 and 422, fix the cause first. Keep the X-Request-ID header for support. The errors and retries guide sets SDK retry counts, and how to handle LLM API 429s has limiter and queue code. The rate limit calculator shows how much traffic fits under a limit.

Questions

Which errors should I retry with backoff?

Rate limits (429), server errors (5xx), timeouts and dropped connections. Do not retry 400, 401, 403, 413 or 422 unchanged; fix the request or the key first.

Why add jitter?

Without it, clients that failed together retry together and overload the server again. Jitter randomises each wait so retries spread out.

Do the OpenAI SDKs already back off?

Yes. The official OpenAI SDKs retry connection errors, 429 and 5xx responses with a short exponential backoff. Set max_retries explicitly, higher for long batch jobs, and add your own limiter on top.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity