Every LLM API has bad minutes: a 503, a timeout, a burst of 429s. Fallback means those minutes don’t become an outage in your app. This post shows how to fail over between OpenAI-compatible providers in your own code. It covers which errors should trigger it, how a circuit breaker keeps it fast, and what has to match between providers so users don’t notice the switch.
One thing first: BroadNet Inference does not route or fail over between providers for you, and neither does BroadRouter. BroadRouter is BroadNet’s own website, not a router. You build fallback yourself, either in your client as shown below or in a gateway such as LiteLLM.
Why keep a fallback provider at all?
Because most self-serve APIs promise nothing about uptime. BroadNet’s terms say it plainly in section 7.4: “self-serve has no availability commitment”, and service levels exist only in a separate written agreement (checked 15 November 2026). The LLM API SLA comparison covers what other providers commit to. If your feature must stay up, a second provider is the practical insurance, whatever the SLA says.
Rate limits are the second reason. A BroadNet pay-as-you-go key allows 30 requests a minute and 4 in flight. A traffic spike above that returns 429s, and a second provider can absorb the overflow while you raise capacity.
Which errors should trigger a failover?
Split errors into two groups. Some are the provider’s fault and may succeed elsewhere. Others are your fault and will fail everywhere.
| Error | Fail over? | Why |
|---|---|---|
| Connection error, timeout | Yes, after retries | The provider or the network is unreachable |
| 500, 502, 503, 504 | Yes, after retries | BroadNet’s docs say to “retry with bounded exponential backoff” first |
| 429, rate or concurrency limit | Yes, after backoff | Capacity at this provider, not a bad request |
| 429, spending limit | No | Failing over bypasses the budget you set on purpose |
| 400, 422 | No | The request is invalid and will fail on the next provider too |
| 401, 402 | No | Key, allowance or balance. On BroadNet a 401 also covers an exhausted pay-as-you-go allowance, so failing over hides a billing problem |
| 403, 413 | No | Policy or size; fix the request |
BroadNet’s API reference says a 429 means “a rate, concurrency, token-rate or spending limit was reached” and to “use response details and bounded backoff”. Read those details before you fail over, so a spending-limit 429 stops the request rather than moving it. The error codes guide lists every status.
Retry before you fail over. The OpenAI Python SDK already retries connection errors, 408, 409, 429 and 5xx responses “2 times by default, with a short exponential backoff”, and its default timeout is 10 minutes (README, checked 15 November 2026). Ten minutes is far too long to wait before failing over. Set timeout per client. The 429 handling post covers backoff with jitter.
What does failover look like in Python?
Two clients, an ordered list and a small circuit breaker. The primary is GLM 5.3 on BroadNet. The backup is the same open-weight model on DeepInfra, whose docs give the base URL https://api.deepinfra.com/v1/openai (checked 15 November 2026).
import os
import time
from openai import (OpenAI, APIConnectionError, APITimeoutError,
RateLimitError, InternalServerError)
PROVIDERS = [
{"name": "broadnet", "model": "broadnet/glm-5.3",
"client": OpenAI(base_url="https://inference.broadnet.ai/v1",
api_key=os.environ["BROADNET_API_KEY"],
max_retries=2, timeout=60)},
{"name": "deepinfra", "model": "zai-org/GLM-5.3",
"client": OpenAI(base_url="https://api.deepinfra.com/v1/openai",
api_key=os.environ["DEEPINFRA_API_KEY"],
max_retries=2, timeout=60)},
]
# Provider-side failures only. 400, 401, 402, 403 and 422 raise straight through.
FAILOVER_ON = (APIConnectionError, APITimeoutError, RateLimitError, InternalServerError)
TRIP_AFTER = 3 # consecutive failures before the breaker opens
COOLDOWN = 30 # seconds to skip a provider once its breaker is open
state = {p["name"]: {"fails": 0, "open_until": 0.0} for p in PROVIDERS}
def chat(messages, **kwargs):
last_error = None
for p in PROVIDERS:
s = state[p["name"]]
if time.monotonic() < s["open_until"]:
continue # breaker open: don't wait on a provider that is down
try:
reply = p["client"].chat.completions.create(
model=p["model"], messages=messages, **kwargs)
s["fails"] = 0
return p["name"], reply
except FAILOVER_ON as err:
last_error = err
s["fails"] += 1
if s["fails"] >= TRIP_AFTER:
s["open_until"] = time.monotonic() + COOLDOWN
raise RuntimeError("every provider failed") from last_error
served_by, reply = chat([{"role": "user", "content": "Reply with one short sentence."}],
max_tokens=1000)
print(served_by, reply.model, reply.choices[0].message.content)
Four details matter:
- Exception classes. The SDK raises
RateLimitErrorfor 429,InternalServerErrorfor 500 and above, andAPIConnectionError(withAPITimeoutErroras a subclass) when it can’t connect. Everything else stays uncaught on purpose. To leave spending-limit 429s out, inspecterr.responsebefore you continue. - The circuit breaker. Without it, every request pays the full timeout on a dead primary before moving on. After three straight failures the primary is skipped for 30 seconds. After that, the next request tests it again.
- Log who served it. Return the provider name and keep
reply.model. When quality shifts, you need to know which provider answered. - Reasoning budget. GLM 5.3 reasons before it answers. BroadNet’s docs say reasoning counts toward
max_tokens, so leave 1,000 or more on every provider.
How do you map model IDs across providers?
The same open-weight model has a different ID at each host, and the features each host lists can differ. GLM 5.3 on 15 November 2026:
| BroadNet Inference | DeepInfra | Fireworks AI | |
|---|---|---|---|
| Model ID | broadnet/glm-5.3 |
zai-org/GLM-5.3 |
accounts/fireworks/models/glm-5p3 |
| Context | 1,048,576 tokens | 1,048,576 tokens | “1040k” tokens |
| Tools | Documented | “Function” listed | Supported |
| Structured output | Documented | “JSON” listed | Not checked |
Two rules follow. First, keep the mapping in config, not scattered through code. Second, match your fallback to what your requests use. On BroadNet, tools and structured output are documented for GLM 5.3 and Qwen3.8 27B only. A tool-calling agent can fail over from GLM 5.3 to GLM 5.3 elsewhere. It cannot safely fail over to a model that doesn’t document tools. If the backup has a smaller context window, trim the prompt or skip that backup for long requests. Base URLs for 14 providers are in the base URL reference.
How do you keep answers comparable after a switch?
The same weights at two hosts can still answer differently, because hosts can serve different precisions, chat templates or default parameters. A different model answers differently for certain.
- Run your eval set against every entry in the list, not only the primary.
- Pin
temperature,max_tokensand the system prompt per request, so provider defaults can’t change the output. - Validate structured output the same way on every path. Fixing invalid JSON covers the repair-and-retry loop.
- Prefer the same model elsewhere over a different model at the same host. Users notice a style change more than a few seconds of latency.
What about streaming and tool calls?
A stream can fail halfway. BroadNet’s docs say a stream that ends before data: [DONE] should be treated as incomplete: record the X-Request-ID and decide whether your application can safely send a new request.
- Before the first token: fail over freely. The user has seen nothing.
- After tokens were shown: restart on the backup and replace the partial answer, or show an error with a retry button. Splicing two providers’ half-answers together reads badly.
- Tool calls with side effects: never re-run a step that sends an email, charges a card or writes to a database just because the model call failed. Give each action an idempotency key. Long-running agents covers checkpoints.
Streaming with SSE explains how to detect a stream that ended early.
What does failover cost?
- Duplicate work. A timeout on your side does not prove the provider stopped. On per-token billing, every retry is a new inference, so expect to pay twice for some requests.
- Different prices. The backup bills at its own rates. DeepInfra listed a promotional price for GLM-5.3 on 15 November 2026, so recheck before you budget. BroadNet charges $0.78 input, $0.173 cached input and $3.46 output per 1M tokens.
- Cold caches. Cached input does not move between providers. The first requests on the backup pay full input price on a prefix that was cached on the primary. See cached input tokens.
- Hedging. Sending every request to two providers at once and keeping the first answer cuts tail latency, but roughly doubles the bill. Use it only where latency is worth that.
Does the fallback meet the same data terms?
A fallback provider is another processor that receives the same prompts. If you promised customers something about their data, the promise has to hold on every entry in the list.
BroadNet’s privacy statement says: “We do not retain prompts, audio, images, answers or transcripts”. Inference content is not retained; necessary account, billing and security metadata is kept as that statement describes. It also says inference data is not used to train, fine-tune or evaluate models. The statement says hosted models “run on BroadNet-owned or leased hardware in the United States, with partner capacity as backup”; DeepSeek V4.1 Flash is partner-hosted. For data with residency requirements, the terms (section 7.3a) say to agree processing locations before you send it. Check the same points for the backup: retention, training use, location and subprocessors. If it falls short, leave sensitive requests off the fallback path and fail them closed. The zero data retention checklist lists the questions.
Should you use a gateway like LiteLLM instead?
Use a gateway if several services need the same fallback rules. LiteLLM’s proxy takes a fallbacks list that is tried in order. num_retries retries a model group before it falls back. allowed_fails and cooldown_time take a failing deployment out for a while, which works as a circuit breaker. Separate context_window_fallbacks and content_policy_fallbacks handle those error types (LiteLLM docs, checked 15 November 2026):
model_list:
- model_name: glm
litellm_params:
model: openai/broadnet/glm-5.3
api_base: https://inference.broadnet.ai/v1
api_key: os.environ/BROADNET_API_KEY
- model_name: glm-backup
litellm_params:
model: openai/zai-org/GLM-5.3
api_base: https://api.deepinfra.com/v1/openai
api_key: os.environ/DEEPINFRA_API_KEY
router_settings:
fallbacks: [{"glm": ["glm-backup"]}]
allowed_fails: 3
cooldown_time: 30
litellm_settings:
num_retries: 2
request_timeout: 60
The LiteLLM proxy guide shows the BroadNet entry in full. Both entries use LiteLLM’s generic openai/ prefix with an api_base, the same pattern the guide uses. A gateway adds a hop and one more thing to keep running. About 40 lines in your client, as above, are enough for a single app.
How do you test failover before you need it?
- Point the primary at an unreachable host in a test config and check that requests land on the backup.
- Send a deliberately invalid request and check that it does not fail over.
- Use an invalid primary key and check that the 401 surfaces as an error, not a silent switch.
- Watch the breaker: after three failures, the primary should be skipped without waiting for the timeout.
- Alert when the backup serves more than a small share of traffic. Silent failover can hide a broken primary for days.
LiteLLM’s Router accepts mock_testing_fallbacks=True in direct tests to force a fallback.
Where BroadNet Inference fits
BroadRouter is BroadNet’s website, and BroadNet Inference can be either entry in your list. As primary, it gives you GLM 5.3 and Qwen3.8 27B with documented tools and structured output, plus Gemma 4 VL 26B and DeepSeek V4.1 Flash, on one base URL with no content retained. As a backup, it is one more OpenAI-compatible endpoint to add to the list. Either way, the failover logic is yours: BroadNet does not route between providers, and self-serve has no SLA. If you need contracted service levels, those come only with a separate written agreement. Compare the models, or create an account and test your chain with the $5 starting credit on Gemma 4 VL 26B. GLM 5.3 and Qwen3.8 27B need a top-up.
