An application that talks to one model needs no router. Routers appear when a team uses several models or providers and wants one place to choose between them.
How it works
The application sends an OpenAI-style request to the router. The router applies its rules, forwards the request to a backend, and returns the answer in the same format. There are three common kinds:
- Hosted aggregators. OpenRouter, for example, offers “hundreds of AI models through a single API endpoint” at
https://openrouter.ai/api/v1, with automatic fallbacks. You hold one key and one bill; the aggregator holds accounts with the providers. - Self-hosted gateways. The LiteLLM Router runs in your own infrastructure. It spreads traffic across deployments (simple shuffle by default, or latency-, usage- or cost-based), retries rate-limit errors with exponential backoff, and puts a deployment in cooldown after a 429 or a high failure rate.
- Model-choice routers. Research such as RouteLLM trains a classifier that sends easy prompts to a cheaper model and hard ones to a stronger model, to cut cost without losing quality.
A self-hosted gateway is configured as a list of backends. This LiteLLM entry adds one OpenAI-compatible provider:
model_list:
- model_name: flash
litellm_params:
model: openai/broadnet/deepseek-v4.1-flash
api_base: https://inference.broadnet.ai/v1
api_key: os.environ/BROADNET_API_KEY
Why it matters
- Reliability. A router can fail over when one provider returns 429 or 5xx errors. See rate limits.
- Cost. Sending simple prompts to a cheaper model can lower the bill.
- One integration. Code talks to one base URL and one key format.
The trade-offs are real. A router adds a network hop. A hosted router is one more party that sees your prompts, so retention depends on both the router and the provider it picks. Models also differ in tokenizers, reasoning behavior and parameter support, so a silent fallback can change answers.
On BroadNet Inference
BroadNet Inference is not a router. It is a single provider with one short model list at https://inference.broadnet.ai/v1; BroadNet’s Terms say its models run on BroadNet-owned or leased US hardware or with US inference partners. There is no automatic routing or fallback across providers. You choose the model in each request, for example broadnet/deepseek-v4.1-flash or broadnet/glm-5.3.
Because the API is OpenAI-compatible, it can sit behind a router as one backend. The LiteLLM proxy guide shows the setup. For how a router and a flat-rate provider compare, read OpenRouter alternative for agents. BroadNet does not retain prompts or outputs; if you add a hosted router in front, check its data retention too.
Common mistakes
- Retrying a 401 or 403 on another provider. Those are key or policy errors; fix them first. See errors and retries.
- Treating model names as interchangeable. Test each fallback model on your own prompts.