BroadRouter
Get started

LLM router

Updated

An LLM router is software or a service that sits between an application and one or more language model APIs and decides, for each request, which model or provider handles it, using rules such as cost, latency, availability or the type of task.

Also called: AI gateway, LLM gateway, model router, LLM aggregator

An application that talks to one model needs no router. Routers appear when a team uses several models or providers and wants one place to choose between them.

How it works

The application sends an OpenAI-style request to the router. The router applies its rules, forwards the request to a backend, and returns the answer in the same format. There are three common kinds:

  • Hosted aggregators. OpenRouter, for example, offers “hundreds of AI models through a single API endpoint” at https://openrouter.ai/api/v1, with automatic fallbacks. You hold one key and one bill; the aggregator holds accounts with the providers.
  • Self-hosted gateways. The LiteLLM Router runs in your own infrastructure. It spreads traffic across deployments (simple shuffle by default, or latency-, usage- or cost-based), retries rate-limit errors with exponential backoff, and puts a deployment in cooldown after a 429 or a high failure rate.
  • Model-choice routers. Research such as RouteLLM trains a classifier that sends easy prompts to a cheaper model and hard ones to a stronger model, to cut cost without losing quality.

A self-hosted gateway is configured as a list of backends. This LiteLLM entry adds one OpenAI-compatible provider:

model_list:
  - model_name: flash
    litellm_params:
      model: openai/broadnet/deepseek-v4.1-flash
      api_base: https://inference.broadnet.ai/v1
      api_key: os.environ/BROADNET_API_KEY

Why it matters

  • Reliability. A router can fail over when one provider returns 429 or 5xx errors. See rate limits.
  • Cost. Sending simple prompts to a cheaper model can lower the bill.
  • One integration. Code talks to one base URL and one key format.

The trade-offs are real. A router adds a network hop. A hosted router is one more party that sees your prompts, so retention depends on both the router and the provider it picks. Models also differ in tokenizers, reasoning behavior and parameter support, so a silent fallback can change answers.

On BroadNet Inference

BroadNet Inference is not a router. It is a single provider with one short model list at https://inference.broadnet.ai/v1; BroadNet’s Terms say its models run on BroadNet-owned or leased US hardware or with US inference partners. There is no automatic routing or fallback across providers. You choose the model in each request, for example broadnet/deepseek-v4.1-flash or broadnet/glm-5.3.

Because the API is OpenAI-compatible, it can sit behind a router as one backend. The LiteLLM proxy guide shows the setup. For how a router and a flat-rate provider compare, read OpenRouter alternative for agents. BroadNet does not retain prompts or outputs; if you add a hosted router in front, check its data retention too.

Common mistakes

  • Retrying a 401 or 403 on another provider. Those are key or policy errors; fix them first. See errors and retries.
  • Treating model names as interchangeable. Test each fallback model on your own prompts.

Questions

Is an LLM router the same as a load balancer?

A load balancer is one part of a router. A router can also pick a different model for a task, fall back to another provider after an error, and translate between API formats. A plain load balancer only spreads traffic across identical servers.

Does a router add latency?

Usually a little, because the request makes one extra network hop and the router may inspect it. Fallbacks can add more, since a failed attempt runs before the retry. Measure time to first token with and without the router.

Is BroadNet Inference an LLM router?

No. Despite this site's name, BroadNet Inference is a single provider; BroadNet's Terms say its models run on BroadNet-owned or leased US hardware or with US inference partners. You pick the model in each request. You can place it behind a router as one of the providers.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity