BroadRouter
Get started

Model fallback

Updated

Model fallback is a reliability pattern where a failed LLM request is retried on a different model or provider from an ordered list. The trigger can be an error, a rate limit, a context-length error or a content-policy refusal. It runs in your client or in a gateway in front of the APIs.

Also called: fallback model, model failover, provider fallback

Every API has bad minutes. Fallback means a bad minute at one provider does not become an outage in your app.

How it works

You keep an ordered list of models. When a call fails, after its own retries, the next one in the list gets the same request.

Two gateways from the sources:

Gateway How fallback works
LiteLLM proxy fallbacks lists model groups, tried “in-order”. Separate context_window_fallbacks and content_policy_fallbacks handle those error types. num_retries retries a group before falling back; allowed_fails and cooldown_time take a failing deployment out for a while.
OpenRouter A models array. “If the first model returns an error, OpenRouter will automatically try the next model in the list.” Any error can trigger it, including context length, moderation, rate limits and downtime. If the last model fails, that error is returned.

A minimal client-side version:

from openai import APITimeoutError, InternalServerError, RateLimitError

CHAIN = [("broadnet", "broadnet/glm-5.3"), ("other", "backup-model")]

def chat_with_fallback(clients, msgs):
    for client_name, model in CHAIN:
        try:
            return clients[client_name].chat.completions.create(model=model, messages=msgs)
        except (RateLimitError, APITimeoutError, InternalServerError):
            continue
    raise RuntimeError("all models failed")

Why it matters

  • Uptime. A second provider covers incidents and rate limits at the first.
  • Different output. The backup model writes differently, follows tools differently and may cost more. Run your eval on every model in the chain, not only the first.
  • Do not fall back on your own bugs. A 400 or 422 means the request is wrong; it will fail on the next model too. Fall back on 429, timeouts and 5xx after bounded exponential backoff.

On BroadNet Inference

BroadNet does not route requests automatically and has no fallback across models. BroadRouter, this site, is BroadNet’s information site and does not route requests either. You implement fallback yourself: in your client as above (the LLM API failover how-to adds a circuit breaker and model ID mapping), or in a gateway; see LLM router and the LiteLLM proxy guide for adding BroadNet as a provider. BroadNet’s API reference says to retry 5xx with bounded exponential backoff and to use response details and bounded backoff on 429. The errors and retries guide lists each status.

Common mistakes

  • Falling back on 401 or 402 errors, which hides a billing or key problem.
  • Not logging which model served each reply, so quality changes are hard to trace.

Questions

Should I retry or fall back first?

Retry first for short faults such as a 503. LiteLLM works this way: it falls back to another model group only after a call fails num_retries times.

Which model am I billed for after a fallback?

On OpenRouter, the one that served the request: it says requests "are priced using the model that was ultimately used", reported in the response model field.

Does BroadNet route or fall back automatically?

No. BroadNet does not offer automatic routing, and BroadRouter is BroadNet's information site, not a router. Build fallback in your client or put a gateway such as LiteLLM in front.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity