---
title: "LLM API failover to a backup provider, in Python | BroadRouter"
description: "How to fail over between OpenAI-compatible LLM providers: which errors to fail over on, a circuit breaker, model ID mapping, streaming, cost and privacy parity."
url: https://broadrouter.com/blog/llm-api-failover-fallback/
lang: en
updated: 2026-11-15
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# LLM API failover: fall back to a second provider in your own code

By the BroadNet Inference team · 8 min read · Published 2026-11-15 · Updated 2026-11-16

Keep an ordered list of OpenAI-compatible providers, each with its own client and model ID. Fail over on connection errors, timeouts, 5xx and rate-limit 429s once the SDK's own retries are spent. Never fail over on 400, 401, 402, 403, 413 or 422. Add a circuit breaker, and check that the backup matches the primary on features and data terms.

## Key takeaways

-   Fail over on connection errors, timeouts, 5xx and rate-limit 429s, after bounded retries. A 400, 401, 402, 403, 413 or 422 will fail the same way on the next provider.
-   Model IDs differ per provider: GLM 5.3 is \`broadnet/glm-5.3\` on BroadNet, \`zai-org/GLM-5.3\` on DeepInfra and \`accounts/fireworks/models/glm-5p3\` on Fireworks.
-   A circuit breaker stops you paying a timeout on every request while the primary is down.
-   Fail over a stream only before the first token, and never re-run a tool call that changes something.
-   BroadNet and BroadRouter do not route or fail over for you. BroadNet self-serve has no availability commitment, so production apps should keep a fallback.

**On this page**

1.  Why keep a fallback provider at all?
2.  Which errors should trigger a failover?
3.  What does failover look like in Python?
4.  How do you map model IDs across providers?
5.  How do you keep answers comparable after a switch?
6.  What about streaming and tool calls?
7.  What does failover cost?
8.  Does the fallback meet the same data terms?
9.  Should you use a gateway like LiteLLM instead?
10.  How do you test failover before you need it?
11.  Where BroadNet Inference fits

Every LLM API has bad minutes: a 503, a timeout, a burst of 429s. Fallback means those minutes don’t become an outage in your app. This post shows how to fail over between [OpenAI-compatible](https://broadrouter.com/glossary/openai-compatible-api/) providers in your own code. It covers which errors should trigger it, how a circuit breaker keeps it fast, and what has to match between providers so users don’t notice the switch.

One thing first: **BroadNet Inference does not route or fail over between providers for you**, and neither does BroadRouter. BroadRouter is BroadNet’s own website, not a router. You build fallback yourself, either in your client as shown below or in a gateway such as LiteLLM.

## Why keep a fallback provider at all?

Because most self-serve APIs promise nothing about uptime. BroadNet’s [terms](https://www.broadnet.ai/terms) say it plainly in section 7.4: “self-serve has no availability commitment”, and service levels exist only in a separate written agreement (checked 15 November 2026). The [LLM API SLA comparison](https://broadrouter.com/blog/llm-api-sla-uptime-compared/) covers what other providers commit to. If your feature must stay up, a second provider is the practical insurance, whatever the SLA says.

[Rate limits](https://broadrouter.com/glossary/rate-limit/) are the second reason. A BroadNet pay-as-you-go key allows 30 requests a minute and 4 in flight. A traffic spike above that returns 429s, and a second provider can absorb the overflow while you raise capacity.

## Which errors should trigger a failover?

Split errors into two groups. Some are the provider’s fault and may succeed elsewhere. Others are your fault and will fail everywhere.

| Error | Fail over? | Why |
| --- | --- | --- |
| Connection error, timeout | Yes, after retries | The provider or the network is unreachable |
| 500, 502, 503, 504 | Yes, after retries | BroadNet’s docs say to “retry with bounded exponential backoff” first |
| 429, rate or concurrency limit | Yes, after backoff | Capacity at this provider, not a bad request |
| 429, spending limit | No | Failing over bypasses the budget you set on purpose |
| 400, 422 | No | The request is invalid and will fail on the next provider too |
| 401, 402 | No | Key, allowance or balance. On BroadNet a 401 also covers an exhausted pay-as-you-go allowance, so failing over hides a billing problem |
| 403, 413 | No | Policy or size; fix the request |

BroadNet’s [API reference](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-llm-api-failover-fallback&utm_content=body) says a 429 means “a rate, concurrency, token-rate or spending limit was reached” and to “use response details and bounded backoff”. Read those details before you fail over, so a spending-limit 429 stops the request rather than moving it. The [error codes guide](https://broadrouter.com/guides/errors-and-retries/) lists every status.

Retry before you fail over. The OpenAI Python SDK already retries connection errors, 408, 409, 429 and 5xx responses “2 times by default, with a short exponential backoff”, and its default timeout is 10 minutes ([README](https://github.com/openai/openai-python), checked 15 November 2026). Ten minutes is far too long to wait before failing over. Set `timeout` per client. The [429 handling post](https://broadrouter.com/blog/handle-llm-api-429-rate-limits/) covers backoff with jitter.

## What does failover look like in Python?

Two clients, an ordered list and a small circuit breaker. The primary is GLM 5.3 on BroadNet. The backup is the same open-weight model on DeepInfra, whose docs give the base URL `https://api.deepinfra.com/v1/openai` (checked 15 November 2026).

```python
import os
import time
from openai import (OpenAI, APIConnectionError, APITimeoutError,
                    RateLimitError, InternalServerError)

PROVIDERS = [
    {"name": "broadnet", "model": "broadnet/glm-5.3",
     "client": OpenAI(base_url="https://inference.broadnet.ai/v1",
                      api_key=os.environ["BROADNET_API_KEY"],
                      max_retries=2, timeout=60)},
    {"name": "deepinfra", "model": "zai-org/GLM-5.3",
     "client": OpenAI(base_url="https://api.deepinfra.com/v1/openai",
                      api_key=os.environ["DEEPINFRA_API_KEY"],
                      max_retries=2, timeout=60)},
]

# Provider-side failures only. 400, 401, 402, 403 and 422 raise straight through.
FAILOVER_ON = (APIConnectionError, APITimeoutError, RateLimitError, InternalServerError)
TRIP_AFTER = 3     # consecutive failures before the breaker opens
COOLDOWN = 30      # seconds to skip a provider once its breaker is open
state = {p["name"]: {"fails": 0, "open_until": 0.0} for p in PROVIDERS}

def chat(messages, **kwargs):
    last_error = None
    for p in PROVIDERS:
        s = state[p["name"]]
        if time.monotonic() < s["open_until"]:
            continue  # breaker open: don't wait on a provider that is down
        try:
            reply = p["client"].chat.completions.create(
                model=p["model"], messages=messages, **kwargs)
            s["fails"] = 0
            return p["name"], reply
        except FAILOVER_ON as err:
            last_error = err
            s["fails"] += 1
            if s["fails"] >= TRIP_AFTER:
                s["open_until"] = time.monotonic() + COOLDOWN
    raise RuntimeError("every provider failed") from last_error

served_by, reply = chat([{"role": "user", "content": "Reply with one short sentence."}],
                        max_tokens=1000)
print(served_by, reply.model, reply.choices[0].message.content)
```

Four details matter:

1.  **Exception classes.** The SDK raises `RateLimitError` for 429, `InternalServerError` for 500 and above, and `APIConnectionError` (with `APITimeoutError` as a subclass) when it can’t connect. Everything else stays uncaught on purpose. To leave spending-limit 429s out, inspect `err.response` before you continue.
2.  **The circuit breaker.** Without it, every request pays the full timeout on a dead primary before moving on. After three straight failures the primary is skipped for 30 seconds. After that, the next request tests it again.
3.  **Log who served it.** Return the provider name and keep `reply.model`. When quality shifts, you need to know which provider answered.
4.  **Reasoning budget.** GLM 5.3 reasons before it answers. BroadNet’s docs say reasoning counts toward `max_tokens`, so leave 1,000 or more on every provider.

## How do you map model IDs across providers?

The same open-weight model has a different ID at each host, and the features each host lists can differ. GLM 5.3 on 15 November 2026:

|  | BroadNet Inference | DeepInfra | Fireworks AI |
| --- | --- | --- | --- |
| Model ID | `broadnet/glm-5.3` | `zai-org/GLM-5.3` | `accounts/fireworks/models/glm-5p3` |
| Context | 1,048,576 tokens | 1,048,576 tokens | “1040k” tokens |
| Tools | Documented | “Function” listed | Supported |
| Structured output | Documented | “JSON” listed | Not checked |

Two rules follow. First, keep the mapping in config, not scattered through code. Second, match your fallback to what your requests use. On BroadNet, tools and structured output are documented for GLM 5.3 and Qwen3.8 27B only. A tool-calling agent can fail over from GLM 5.3 to GLM 5.3 elsewhere. It cannot safely fail over to a model that doesn’t document tools. If the backup has a smaller context window, trim the prompt or skip that backup for long requests. Base URLs for 14 providers are in the [base URL reference](https://broadrouter.com/blog/openai-compatible-api-base-urls/).

## How do you keep answers comparable after a switch?

The same weights at two hosts can still answer differently, because hosts can serve different precisions, chat templates or default parameters. A different model answers differently for certain.

-   Run your [eval set](https://broadrouter.com/blog/evaluate-llm-before-switching/) against every entry in the list, not only the primary.
-   Pin `temperature`, `max_tokens` and the system prompt per request, so provider defaults can’t change the output.
-   Validate structured output the same way on every path. [Fixing invalid JSON](https://broadrouter.com/blog/fix-invalid-json-from-llm/) covers the repair-and-retry loop.
-   Prefer the same model elsewhere over a different model at the same host. Users notice a style change more than a few seconds of latency.

## What about streaming and tool calls?

A stream can fail halfway. BroadNet’s docs say a stream that ends before `data: [DONE]` should be treated as incomplete: record the `X-Request-ID` and decide whether your application can safely send a new request.

-   **Before the first token:** fail over freely. The user has seen nothing.
-   **After tokens were shown:** restart on the backup and replace the partial answer, or show an error with a retry button. Splicing two providers’ half-answers together reads badly.
-   **Tool calls with side effects:** never re-run a step that sends an email, charges a card or writes to a database just because the model call failed. Give each action an [idempotency key](https://broadrouter.com/glossary/idempotency-key/). [Long-running agents](https://broadrouter.com/blog/long-running-agents-timeouts-retries/) covers checkpoints.

[Streaming with SSE](https://broadrouter.com/blog/llm-streaming-sse-explained/) explains how to detect a stream that ended early.

## What does failover cost?

-   **Duplicate work.** A timeout on your side does not prove the provider stopped. On per-token billing, every retry is a new inference, so expect to pay twice for some requests.
-   **Different prices.** The backup bills at its own rates. DeepInfra listed a promotional price for GLM-5.3 on 15 November 2026, so recheck before you budget. BroadNet charges $0.78 input, $0.173 cached input and $3.46 output per 1M tokens.
-   **Cold caches.** Cached input does not move between providers. The first requests on the backup pay full input price on a prefix that was cached on the primary. See [cached input tokens](https://broadrouter.com/blog/cached-input-tokens-explained/).
-   **Hedging.** Sending every request to two providers at once and keeping the first answer cuts tail latency, but roughly doubles the bill. Use it only where latency is worth that.

## Does the fallback meet the same data terms?

A fallback provider is another processor that receives the same prompts. If you promised customers something about their data, the promise has to hold on every entry in the list.

BroadNet’s [privacy statement](https://www.broadnet.ai/inference-privacy) says: “We do not retain prompts, audio, images, answers or transcripts”. Inference content is not retained; necessary account, billing and security metadata is kept as that statement describes. It also says inference data is not used to train, fine-tune or evaluate models. The statement says hosted models “run on BroadNet-owned or leased hardware in the United States, with partner capacity as backup”; DeepSeek V4.1 Flash is partner-hosted. For data with residency requirements, the terms (section 7.3a) say to agree processing locations before you send it. Check the same points for the backup: retention, training use, location and subprocessors. If it falls short, leave sensitive requests off the fallback path and fail them closed. The [zero data retention checklist](https://broadrouter.com/blog/zero-data-retention-llm-api-checklist/) lists the questions.

## Should you use a gateway like LiteLLM instead?

Use a gateway if several services need the same fallback rules. LiteLLM’s proxy takes a `fallbacks` list that is tried in order. `num_retries` retries a model group before it falls back. `allowed_fails` and `cooldown_time` take a failing deployment out for a while, which works as a circuit breaker. Separate `context_window_fallbacks` and `content_policy_fallbacks` handle those error types ([LiteLLM docs](https://docs.litellm.ai/docs/proxy/reliability), checked 15 November 2026):

```yaml
model_list:
  - model_name: glm
    litellm_params:
      model: openai/broadnet/glm-5.3
      api_base: https://inference.broadnet.ai/v1
      api_key: os.environ/BROADNET_API_KEY
  - model_name: glm-backup
    litellm_params:
      model: openai/zai-org/GLM-5.3
      api_base: https://api.deepinfra.com/v1/openai
      api_key: os.environ/DEEPINFRA_API_KEY

router_settings:
  fallbacks: [{"glm": ["glm-backup"]}]
  allowed_fails: 3
  cooldown_time: 30

litellm_settings:
  num_retries: 2
  request_timeout: 60
```

The [LiteLLM proxy guide](https://broadrouter.com/guides/litellm-proxy-custom-provider/) shows the BroadNet entry in full. Both entries use LiteLLM’s generic `openai/` prefix with an `api_base`, the same pattern the guide uses. A gateway adds a hop and one more thing to keep running. About 40 lines in your client, as above, are enough for a single app.

## How do you test failover before you need it?

1.  Point the primary at an unreachable host in a test config and check that requests land on the backup.
2.  Send a deliberately invalid request and check that it does **not** fail over.
3.  Use an invalid primary key and check that the 401 surfaces as an error, not a silent switch.
4.  Watch the breaker: after three failures, the primary should be skipped without waiting for the timeout.
5.  Alert when the backup serves more than a small share of traffic. Silent failover can hide a broken primary for days.

LiteLLM’s `Router` accepts `mock_testing_fallbacks=True` in direct tests to force a fallback.

## Where BroadNet Inference fits

BroadRouter is BroadNet’s website, and BroadNet Inference can be either entry in your list. As primary, it gives you GLM 5.3 and Qwen3.8 27B with documented tools and structured output, plus Gemma 4 VL 26B and DeepSeek V4.1 Flash, on one [base URL](https://broadrouter.com/glossary/base-url/) with no content retained. As a backup, it is one more OpenAI-compatible endpoint to add to the list. Either way, the failover logic is yours: BroadNet does not route between providers, and self-serve has no SLA. If you need contracted service levels, those come only with a separate written agreement. Compare the [models](https://broadrouter.com/models/), or [create an account](https://inference.broadnet.ai/?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-llm-api-failover-fallback&utm_content=body) and test your chain with the $5 starting credit on Gemma 4 VL 26B. GLM 5.3 and Qwen3.8 27B need a top-up.

Free tools for this

-   [OpenAI to BroadNet migration helper](https://broadrouter.com/tools/openai-to-broadnet-migration/) Paste OpenAI SDK code; get base URL, key and model edits, a diff and the checks that remain.
-   [max\_tokens calculator for reasoning models](https://broadrouter.com/tools/reasoning-token-budget/) Answer length + reasoning budget + margin = a max\_tokens that does not cut the answer off.
-   [JSON schema generator for LLM structured output](https://broadrouter.com/tools/json-schema-generator/) Sample JSON to strict schema, Pydantic, Zod and a ready request.
-   [LLM JSON output validator](https://broadrouter.com/tools/llm-json-output-validator/) Parse errors by line and column, safe repairs, cut-off detection and JSON Schema checks for model output.

[All free tools](https://broadrouter.com/tools/)

## Terms in this article

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[Base URL](https://broadrouter.com/glossary/base-url/)

A base URL is the fixed root address of an API, such as https://api.example.com/v1, that a client puts in front of every endpoint path, such as /chat/completions. In OpenAI-compatible SDKs, changing the base URL, together with the API key and model ID, points the same code at a different provider.

[Concurrency](https://broadrouter.com/glossary/concurrency/)

Concurrency is the number of requests a client has in flight at the same time. LLM APIs often cap it per key or plan, separately from requests per minute, because each open request holds GPU capacity until its response has finished.

[Pay-as-you-go](https://broadrouter.com/glossary/pay-as-you-go/)

Pay-as-you-go is a pricing model in which you pay only for what you use, metered per unit such as tokens, requests or audio minutes, with no subscription or minimum commitment. Most LLM APIs bill this way, from a prepaid balance or a periodic invoice.

[Structured output](https://broadrouter.com/glossary/structured-output/)

Structured output is an LLM API feature that constrains a model's reply to a format the developer specifies, usually a JSON Schema, so code can parse it reliably. It is stricter than JSON mode, which guarantees valid JSON but not a particular shape.

[Tool calling](https://broadrouter.com/glossary/tool-calling/)

Tool calling, also called function calling, is an LLM API feature in which the model returns a structured request to run a function the developer has defined, with JSON arguments. The application runs the function and sends the result back; the model never runs code itself.

[Latency](https://broadrouter.com/glossary/latency/)

Latency, for a language model API, is the time a request takes from being sent to the final output token being received. It is the sum of time to first token and generation time, and is usually reported as percentiles such as p50 and p95 rather than as an average.

[SLA (service level agreement)](https://broadrouter.com/glossary/sla/)

An SLA (service level agreement) is a contractual commitment from a provider to a measurable level of service, usually a monthly uptime percentage, with a defined remedy, usually service credits, if it is missed. An SLA defines what counts as downtime, what is excluded and how to claim.

## Questions

### Which errors should trigger an LLM API fallback?

Connection errors, timeouts, 5xx and rate-limit 429s, after the SDK's own retries. Do not fail over on 400, 401, 402, 403, 413 or 422: the request or the account is wrong, and the next provider fails the same way or hides a billing problem.

### Does BroadNet fail over to another provider automatically?

No. BroadNet Inference does not route requests between models or providers, and BroadRouter, this site, is BroadNet's information site, not a router. You build fallback in your client or in a gateway such as LiteLLM.

### Can I use the same model ID on every provider?

No. Each provider names the same open-weight model differently. GLM 5.3 is \`broadnet/glm-5.3\` on BroadNet, \`zai-org/GLM-5.3\` on DeepInfra and \`accounts/fireworks/models/glm-5p3\` on Fireworks (checked 15 November 2026). Keep a per-provider mapping.

### Do failed attempts cost money?

They can. A timeout on your side does not prove the provider stopped work, and on per-token billing every retry is a new inference. Budget for some double-billed requests and cap attempts per request.

### Should a fallback provider have the same privacy terms?

Yes. A fallback is another processor that sees the same prompts. Check its retention, training and processing-location terms before you add it, or your privacy promise only holds while the primary is up.

### Does BroadNet have an SLA?

Not for self-serve. BroadNet's terms (section 7.4) say self-serve has no availability commitment; service levels exist only in a separate written agreement.

## Keep reading

-   [How to handle LLM API 429 errors: backoff, jitter and limiters](https://broadrouter.com/blog/handle-llm-api-429-rate-limits/)
-   [Long-running agents: timeouts, retries and checkpoints that hold](https://broadrouter.com/blog/long-running-agents-timeouts-retries/)
-   [OpenAI-compatible API base URLs: 14 providers, checked](https://broadrouter.com/blog/openai-compatible-api-base-urls/)
-   [LLM API SLAs compared: what Azure, Bedrock, Google and OpenAI promise](https://broadrouter.com/blog/llm-api-sla-uptime-compared/)
-   [How to evaluate an LLM before switching providers](https://broadrouter.com/blog/evaluate-llm-before-switching/)
-   [A private endpoint starts with a clear workload](https://broadrouter.com/blog/private-endpoint-planning/)
-   [Before fine-tuning: define the problem to solve](https://broadrouter.com/blog/fine-tuning-readiness/)
-   [Guide: BroadNet Inference error codes and when to retry](https://broadrouter.com/guides/errors-and-retries/)
-   [Guide: Route LiteLLM to BroadNet Inference (SDK and proxy)](https://broadrouter.com/guides/litellm-proxy-custom-provider/)
-   [Guide: Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-llm-api-failover-fallback&utm_content=page-cta) ·

**Sources**

-   [BroadNet Inference API reference: errors and retries, limits, streaming, model capabilities (checked 2026-11-15)](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-llm-api-failover-fallback&utm_content=sources)
-   [BroadNet terms of service, sections 7.3a (processing locations) and 7.4 (availability) (checked 2026-11-15)](https://www.broadnet.ai/terms)
-   [BroadNet inference privacy statement: no content retention, no training, US hardware with partner backup (checked 2026-11-15)](https://www.broadnet.ai/inference-privacy)
-   [OpenAI Python SDK README: retries, timeouts and error classes (checked 2026-11-15)](https://github.com/openai/openai-python)
-   [LiteLLM docs, proxy reliability: fallbacks, num\_retries, allowed\_fails, cooldown\_time (checked 2026-11-15)](https://docs.litellm.ai/docs/proxy/reliability)
-   [DeepInfra docs, chat overview: OpenAI-compatible base URL (checked 2026-11-15)](https://docs.deepinfra.com/chat/overview)
-   [DeepInfra model page, zai-org/GLM-5.3: context, function calling and JSON support (checked 2026-11-15)](https://deepinfra.com/zai-org/GLM-5.3)
-   [Fireworks AI model page, GLM 5.3: model ID, context and tool support (checked 2026-11-15)](https://fireworks.ai/models/fireworks/glm-5p3)
