BroadRouter

Use LlamaIndex with BroadNet Inference through OpenAILike

Updated

Install llama-index-llms-openai-like and create OpenAILike with model set to a BroadNet ID such as broadnet/glm-5.3, api_base set to https://inference.broadnet.ai/v1, your BroadNet key, context_window set to the model's real context, is_chat_model=True, and is_function_calling_model=True only for GLM 5.3 or Qwen3.8 27B. Set it as Settings.llm to use it everywhere.

LlamaIndex is a Python framework for RAG and agents. For any OpenAI-compatible API that isn’t OpenAI itself, it provides the OpenAILike class. BroadNet Inference serves standard chat completions at https://inference.broadnet.ai/v1. BroadRouter is BroadNet’s own site; this follows the LlamaIndex API reference for llama-index-llms-openai-like 0.9.0.

Install

pip install llama-index-llms-openai-like
export BROADNET_API_KEY='your-broadnet-key'

Connect and test

import os
from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    model="broadnet/glm-5.3",
    api_base="https://inference.broadnet.ai/v1",
    api_key=os.environ["BROADNET_API_KEY"],
    context_window=1048576,
    is_chat_model=True,
    is_function_calling_model=True,
    max_tokens=2000,  # reasoning counts toward this limit
)

print(llm.complete("Name three uses for a vector index."))

A printed answer means the key, URL and model ID work.

The four settings that matter

Parameter Set it to Default if you don’t
is_chat_model True False, which uses the legacy completions endpoint; BroadNet serves chat completions
context_window The model’s real context 3,900 tokens
is_function_calling_model True only for GLM 5.3 or Qwen3.8 27B False
max_tokens 1,000 or more None (not sent, so the server decides)

is_chat_model matters most. The API reference says it “controls whether the model uses the chat or completion endpoint” and defaults to false. BroadNet’s docs list /v1/chat/completions, not the older completions endpoint.

Use it across an app

from llama_index.core import Settings

Settings.llm = llm

Query engines, chat engines and agents then use BroadNet. Embeddings are separate: BroadNet’s API reference doesn’t list an embeddings endpoint, so keep a different embedding model for indexing.

Context windows

Model ID context_window Tools
GLM 5.3 broadnet/glm-5.3 1048576 Yes
DeepSeek V4.1 Flash broadnet/deepseek-v4.1-flash 1048576 Not listed
Qwen3.8 27B broadnet/qwen3.8-27b 262144 Yes
BroadNet Gemma 4 VL 26B broadnet/gemma-4-26b-a4b 32768 Not listed

A 1M-token context lets a query engine pass many more retrieved chunks per call. That isn’t always better; see long context vs RAG.

Troubleshooting

  • 401. The key is wrong, expired or out of allowance.
  • 404. api_base must end in /v1, and is_chat_model must be True. Then check the model ID. Paid models need a top-up; the $5 credit covers only broadnet/gemma-4-26b-a4b and the two Transcribe models.
  • Empty answers. Reasoning used the output budget. Raise max_tokens. Why answers come back empty.
  • The agent never calls tools. Set is_function_calling_model=True and use broadnet/glm-5.3 or broadnet/qwen3.8-27b.
  • 429 during ingestion or batch queries. A pay-as-you-go key allows 30 requests a minute and 4 in flight. OpenAILike retries 3 times by default (max_retries). See errors and retries.

Questions

Why not use LlamaIndex's OpenAI class with a different api_base?

LlamaIndex's own OpenAI page says to use the OpenAILike integration for any OpenAI-compatible API that is not the official OpenAI API. OpenAILike also lets you declare the context window and tool support yourself.

What happens if I leave context_window out?

OpenAILike defaults to 3,900 tokens, so LlamaIndex trims prompts and retrieved chunks far below what BroadNet's models accept. Set 1048576 for GLM 5.3 and DeepSeek V4.1 Flash, or 262144 for Qwen3.8 27B.

Can a LlamaIndex FunctionAgent use BroadNet?

Yes, with is_function_calling_model=True on a model BroadNet lists with tools, GLM 5.3 or Qwen3.8 27B. Leave it False for DeepSeek V4.1 Flash, where BroadNet's docs don't list tool calling.

Does BroadNet provide embeddings for a LlamaIndex index?

BroadNet's API reference lists chat completions and audio transcriptions, not embeddings. Use a separate embedding model for the index and BroadNet for the LLM.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity