Every API has bad minutes. Fallback means a bad minute at one provider does not become an outage in your app.
How it works
You keep an ordered list of models. When a call fails, after its own retries, the next one in the list gets the same request.
Two gateways from the sources:
| Gateway | How fallback works |
|---|---|
| LiteLLM proxy | fallbacks lists model groups, tried “in-order”. Separate context_window_fallbacks and content_policy_fallbacks handle those error types. num_retries retries a group before falling back; allowed_fails and cooldown_time take a failing deployment out for a while. |
| OpenRouter | A models array. “If the first model returns an error, OpenRouter will automatically try the next model in the list.” Any error can trigger it, including context length, moderation, rate limits and downtime. If the last model fails, that error is returned. |
A minimal client-side version:
from openai import APITimeoutError, InternalServerError, RateLimitError
CHAIN = [("broadnet", "broadnet/glm-5.3"), ("other", "backup-model")]
def chat_with_fallback(clients, msgs):
for client_name, model in CHAIN:
try:
return clients[client_name].chat.completions.create(model=model, messages=msgs)
except (RateLimitError, APITimeoutError, InternalServerError):
continue
raise RuntimeError("all models failed")
Why it matters
- Uptime. A second provider covers incidents and rate limits at the first.
- Different output. The backup model writes differently, follows tools differently and may cost more. Run your eval on every model in the chain, not only the first.
- Do not fall back on your own bugs. A 400 or 422 means the request is wrong; it will fail on the next model too. Fall back on 429, timeouts and 5xx after bounded exponential backoff.
On BroadNet Inference
BroadNet does not route requests automatically and has no fallback across models. BroadRouter, this site, is BroadNet’s information site and does not route requests either. You implement fallback yourself: in your client as above (the LLM API failover how-to adds a circuit breaker and model ID mapping), or in a gateway; see LLM router and the LiteLLM proxy guide for adding BroadNet as a provider. BroadNet’s API reference says to retry 5xx with bounded exponential backoff and to use response details and bounded backoff on 429. The errors and retries guide lists each status.
Common mistakes
- Falling back on 401 or 402 errors, which hides a billing or key problem.
- Not logging which model served each reply, so quality changes are hard to trace.