BroadRouter
Get started

LLM API failover: fall back to a second provider in your own code

By the BroadNet Inference team · 8 min read · Published · Updated

Keep an ordered list of OpenAI-compatible providers, each with its own client and model ID. Fail over on connection errors, timeouts, 5xx and rate-limit 429s once the SDK's own retries are spent. Never fail over on 400, 401, 402, 403, 413 or 422. Add a circuit breaker, and check that the backup matches the primary on features and data terms.

Every LLM API has bad minutes: a 503, a timeout, a burst of 429s. Fallback means those minutes don’t become an outage in your app. This post shows how to fail over between OpenAI-compatible providers in your own code. It covers which errors should trigger it, how a circuit breaker keeps it fast, and what has to match between providers so users don’t notice the switch.

One thing first: BroadNet Inference does not route or fail over between providers for you, and neither does BroadRouter. BroadRouter is BroadNet’s own website, not a router. You build fallback yourself, either in your client as shown below or in a gateway such as LiteLLM.

Why keep a fallback provider at all?

Because most self-serve APIs promise nothing about uptime. BroadNet’s terms say it plainly in section 7.4: “self-serve has no availability commitment”, and service levels exist only in a separate written agreement (checked 15 November 2026). The LLM API SLA comparison covers what other providers commit to. If your feature must stay up, a second provider is the practical insurance, whatever the SLA says.

Rate limits are the second reason. A BroadNet pay-as-you-go key allows 30 requests a minute and 4 in flight. A traffic spike above that returns 429s, and a second provider can absorb the overflow while you raise capacity.

Which errors should trigger a failover?

Split errors into two groups. Some are the provider’s fault and may succeed elsewhere. Others are your fault and will fail everywhere.

Error Fail over? Why
Connection error, timeout Yes, after retries The provider or the network is unreachable
500, 502, 503, 504 Yes, after retries BroadNet’s docs say to “retry with bounded exponential backoff” first
429, rate or concurrency limit Yes, after backoff Capacity at this provider, not a bad request
429, spending limit No Failing over bypasses the budget you set on purpose
400, 422 No The request is invalid and will fail on the next provider too
401, 402 No Key, allowance or balance. On BroadNet a 401 also covers an exhausted pay-as-you-go allowance, so failing over hides a billing problem
403, 413 No Policy or size; fix the request

BroadNet’s API reference says a 429 means “a rate, concurrency, token-rate or spending limit was reached” and to “use response details and bounded backoff”. Read those details before you fail over, so a spending-limit 429 stops the request rather than moving it. The error codes guide lists every status.

Retry before you fail over. The OpenAI Python SDK already retries connection errors, 408, 409, 429 and 5xx responses “2 times by default, with a short exponential backoff”, and its default timeout is 10 minutes (README, checked 15 November 2026). Ten minutes is far too long to wait before failing over. Set timeout per client. The 429 handling post covers backoff with jitter.

What does failover look like in Python?

Two clients, an ordered list and a small circuit breaker. The primary is GLM 5.3 on BroadNet. The backup is the same open-weight model on DeepInfra, whose docs give the base URL https://api.deepinfra.com/v1/openai (checked 15 November 2026).

import os
import time
from openai import (OpenAI, APIConnectionError, APITimeoutError,
                    RateLimitError, InternalServerError)

PROVIDERS = [
    {"name": "broadnet", "model": "broadnet/glm-5.3",
     "client": OpenAI(base_url="https://inference.broadnet.ai/v1",
                      api_key=os.environ["BROADNET_API_KEY"],
                      max_retries=2, timeout=60)},
    {"name": "deepinfra", "model": "zai-org/GLM-5.3",
     "client": OpenAI(base_url="https://api.deepinfra.com/v1/openai",
                      api_key=os.environ["DEEPINFRA_API_KEY"],
                      max_retries=2, timeout=60)},
]

# Provider-side failures only. 400, 401, 402, 403 and 422 raise straight through.
FAILOVER_ON = (APIConnectionError, APITimeoutError, RateLimitError, InternalServerError)
TRIP_AFTER = 3     # consecutive failures before the breaker opens
COOLDOWN = 30      # seconds to skip a provider once its breaker is open
state = {p["name"]: {"fails": 0, "open_until": 0.0} for p in PROVIDERS}


def chat(messages, **kwargs):
    last_error = None
    for p in PROVIDERS:
        s = state[p["name"]]
        if time.monotonic() < s["open_until"]:
            continue  # breaker open: don't wait on a provider that is down
        try:
            reply = p["client"].chat.completions.create(
                model=p["model"], messages=messages, **kwargs)
            s["fails"] = 0
            return p["name"], reply
        except FAILOVER_ON as err:
            last_error = err
            s["fails"] += 1
            if s["fails"] >= TRIP_AFTER:
                s["open_until"] = time.monotonic() + COOLDOWN
    raise RuntimeError("every provider failed") from last_error


served_by, reply = chat([{"role": "user", "content": "Reply with one short sentence."}],
                        max_tokens=1000)
print(served_by, reply.model, reply.choices[0].message.content)

Four details matter:

  1. Exception classes. The SDK raises RateLimitError for 429, InternalServerError for 500 and above, and APIConnectionError (with APITimeoutError as a subclass) when it can’t connect. Everything else stays uncaught on purpose. To leave spending-limit 429s out, inspect err.response before you continue.
  2. The circuit breaker. Without it, every request pays the full timeout on a dead primary before moving on. After three straight failures the primary is skipped for 30 seconds. After that, the next request tests it again.
  3. Log who served it. Return the provider name and keep reply.model. When quality shifts, you need to know which provider answered.
  4. Reasoning budget. GLM 5.3 reasons before it answers. BroadNet’s docs say reasoning counts toward max_tokens, so leave 1,000 or more on every provider.

How do you map model IDs across providers?

The same open-weight model has a different ID at each host, and the features each host lists can differ. GLM 5.3 on 15 November 2026:

BroadNet Inference DeepInfra Fireworks AI
Model ID broadnet/glm-5.3 zai-org/GLM-5.3 accounts/fireworks/models/glm-5p3
Context 1,048,576 tokens 1,048,576 tokens “1040k” tokens
Tools Documented “Function” listed Supported
Structured output Documented “JSON” listed Not checked

Two rules follow. First, keep the mapping in config, not scattered through code. Second, match your fallback to what your requests use. On BroadNet, tools and structured output are documented for GLM 5.3 and Qwen3.8 27B only. A tool-calling agent can fail over from GLM 5.3 to GLM 5.3 elsewhere. It cannot safely fail over to a model that doesn’t document tools. If the backup has a smaller context window, trim the prompt or skip that backup for long requests. Base URLs for 14 providers are in the base URL reference.

How do you keep answers comparable after a switch?

The same weights at two hosts can still answer differently, because hosts can serve different precisions, chat templates or default parameters. A different model answers differently for certain.

  • Run your eval set against every entry in the list, not only the primary.
  • Pin temperature, max_tokens and the system prompt per request, so provider defaults can’t change the output.
  • Validate structured output the same way on every path. Fixing invalid JSON covers the repair-and-retry loop.
  • Prefer the same model elsewhere over a different model at the same host. Users notice a style change more than a few seconds of latency.

What about streaming and tool calls?

A stream can fail halfway. BroadNet’s docs say a stream that ends before data: [DONE] should be treated as incomplete: record the X-Request-ID and decide whether your application can safely send a new request.

  • Before the first token: fail over freely. The user has seen nothing.
  • After tokens were shown: restart on the backup and replace the partial answer, or show an error with a retry button. Splicing two providers’ half-answers together reads badly.
  • Tool calls with side effects: never re-run a step that sends an email, charges a card or writes to a database just because the model call failed. Give each action an idempotency key. Long-running agents covers checkpoints.

Streaming with SSE explains how to detect a stream that ended early.

What does failover cost?

  • Duplicate work. A timeout on your side does not prove the provider stopped. On per-token billing, every retry is a new inference, so expect to pay twice for some requests.
  • Different prices. The backup bills at its own rates. DeepInfra listed a promotional price for GLM-5.3 on 15 November 2026, so recheck before you budget. BroadNet charges $0.78 input, $0.173 cached input and $3.46 output per 1M tokens.
  • Cold caches. Cached input does not move between providers. The first requests on the backup pay full input price on a prefix that was cached on the primary. See cached input tokens.
  • Hedging. Sending every request to two providers at once and keeping the first answer cuts tail latency, but roughly doubles the bill. Use it only where latency is worth that.

Does the fallback meet the same data terms?

A fallback provider is another processor that receives the same prompts. If you promised customers something about their data, the promise has to hold on every entry in the list.

BroadNet’s privacy statement says: “We do not retain prompts, audio, images, answers or transcripts”. Inference content is not retained; necessary account, billing and security metadata is kept as that statement describes. It also says inference data is not used to train, fine-tune or evaluate models. The statement says hosted models “run on BroadNet-owned or leased hardware in the United States, with partner capacity as backup”; DeepSeek V4.1 Flash is partner-hosted. For data with residency requirements, the terms (section 7.3a) say to agree processing locations before you send it. Check the same points for the backup: retention, training use, location and subprocessors. If it falls short, leave sensitive requests off the fallback path and fail them closed. The zero data retention checklist lists the questions.

Should you use a gateway like LiteLLM instead?

Use a gateway if several services need the same fallback rules. LiteLLM’s proxy takes a fallbacks list that is tried in order. num_retries retries a model group before it falls back. allowed_fails and cooldown_time take a failing deployment out for a while, which works as a circuit breaker. Separate context_window_fallbacks and content_policy_fallbacks handle those error types (LiteLLM docs, checked 15 November 2026):

model_list:
  - model_name: glm
    litellm_params:
      model: openai/broadnet/glm-5.3
      api_base: https://inference.broadnet.ai/v1
      api_key: os.environ/BROADNET_API_KEY
  - model_name: glm-backup
    litellm_params:
      model: openai/zai-org/GLM-5.3
      api_base: https://api.deepinfra.com/v1/openai
      api_key: os.environ/DEEPINFRA_API_KEY

router_settings:
  fallbacks: [{"glm": ["glm-backup"]}]
  allowed_fails: 3
  cooldown_time: 30

litellm_settings:
  num_retries: 2
  request_timeout: 60

The LiteLLM proxy guide shows the BroadNet entry in full. Both entries use LiteLLM’s generic openai/ prefix with an api_base, the same pattern the guide uses. A gateway adds a hop and one more thing to keep running. About 40 lines in your client, as above, are enough for a single app.

How do you test failover before you need it?

  1. Point the primary at an unreachable host in a test config and check that requests land on the backup.
  2. Send a deliberately invalid request and check that it does not fail over.
  3. Use an invalid primary key and check that the 401 surfaces as an error, not a silent switch.
  4. Watch the breaker: after three failures, the primary should be skipped without waiting for the timeout.
  5. Alert when the backup serves more than a small share of traffic. Silent failover can hide a broken primary for days.

LiteLLM’s Router accepts mock_testing_fallbacks=True in direct tests to force a fallback.

Where BroadNet Inference fits

BroadRouter is BroadNet’s website, and BroadNet Inference can be either entry in your list. As primary, it gives you GLM 5.3 and Qwen3.8 27B with documented tools and structured output, plus Gemma 4 VL 26B and DeepSeek V4.1 Flash, on one base URL with no content retained. As a backup, it is one more OpenAI-compatible endpoint to add to the list. Either way, the failover logic is yours: BroadNet does not route between providers, and self-serve has no SLA. If you need contracted service levels, those come only with a separate written agreement. Compare the models, or create an account and test your chain with the $5 starting credit on Gemma 4 VL 26B. GLM 5.3 and Qwen3.8 27B need a top-up.

Questions

Which errors should trigger an LLM API fallback?

Connection errors, timeouts, 5xx and rate-limit 429s, after the SDK's own retries. Do not fail over on 400, 401, 402, 403, 413 or 422: the request or the account is wrong, and the next provider fails the same way or hides a billing problem.

Does BroadNet fail over to another provider automatically?

No. BroadNet Inference does not route requests between models or providers, and BroadRouter, this site, is BroadNet's information site, not a router. You build fallback in your client or in a gateway such as LiteLLM.

Can I use the same model ID on every provider?

No. Each provider names the same open-weight model differently. GLM 5.3 is `broadnet/glm-5.3` on BroadNet, `zai-org/GLM-5.3` on DeepInfra and `accounts/fireworks/models/glm-5p3` on Fireworks (checked 15 November 2026). Keep a per-provider mapping.

Do failed attempts cost money?

They can. A timeout on your side does not prove the provider stopped work, and on per-token billing every retry is a new inference. Budget for some double-billed requests and cap attempts per request.

Should a fallback provider have the same privacy terms?

Yes. A fallback is another processor that sees the same prompts. Check its retention, training and processing-location terms before you add it, or your privacy promise only holds while the primary is up.

Does BroadNet have an SLA?

Not for self-serve. BroadNet's terms (section 7.4) say self-serve has no availability commitment; service levels exist only in a separate written agreement.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity