Haystack is deepset’s open-source framework for RAG pipelines and agents. Its OpenAIChatGenerator talks to any OpenAI-compatible server through api_base_url. BroadNet Inference serves standard chat completions at https://inference.broadnet.ai/v1, so that is the component to use. BroadRouter is BroadNet’s own site; this follows the Haystack 3.5 docs.
Install
pip install haystack-ai
export BROADNET_API_KEY='your-broadnet-key'
Connect
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret
generator = OpenAIChatGenerator(
api_base_url="https://inference.broadnet.ai/v1",
api_key=Secret.from_env_var("BROADNET_API_KEY"),
model="broadnet/glm-5.3",
generation_kwargs={"max_tokens": 4000},
)
reply = generator.run([ChatMessage.from_user("Name three uses of a RAG pipeline.")])
print(reply["replies"][0].text)
modelis sent unchanged. Use the BroadNet ID with itsbroadnet/prefix.api_keyis a HaystackSecret. Without it, Haystack looks forOPENAI_API_KEY.generation_kwargsare passed into the chat completion request. Setmax_tokenshigh enough for reasoning (see below).
Stream the answer
from haystack.components.generators.utils import print_streaming_chunk
generator = OpenAIChatGenerator(
api_base_url="https://inference.broadnet.ai/v1",
api_key=Secret.from_env_var("BROADNET_API_KEY"),
model="broadnet/glm-5.3",
streaming_callback=print_streaming_chunk,
)
Haystack then sends stream: true with stream_options.include_usage, so token counts arrive in the last chunk. BroadNet’s docs describe the same option.
Tools and agents
from haystack.components.agents import Agent
from haystack.tools import Tool
def get_weather(city: str) -> str:
return f"21C in {city}"
weather = Tool(
name="get_weather",
description="Get the weather for a city",
parameters={"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]},
function=get_weather,
)
chat = OpenAIChatGenerator(
api_base_url="https://inference.broadnet.ai/v1",
api_key=Secret.from_env_var("BROADNET_API_KEY"),
model="broadnet/glm-5.3",
tools=[weather],
)
agent = Agent(chat_generator=chat, tools=[weather])
result = agent.run(messages=[ChatMessage.from_user("What's the weather in Riyadh?")])
print(result["messages"][-1].text)
BroadNet’s docs list tools and structured output for broadnet/glm-5.3 and broadnet/qwen3.8-27b only. Keep tools_strict at its default of False unless you have tested strict schemas.
Which model to pick
| Pipeline | Model |
|---|---|
| Agents and tool calls | broadnet/glm-5.3 |
| RAG over scanned pages or Arabic text, with tools | broadnet/qwen3.8-27b |
| Long-context answering on a flat-price plan | broadnet/deepseek-v4.1-flash |
| Trying it out on the free starting credit | broadnet/gemma-4-26b-a4b (32,768-token context) |
Retrieved context can be large. GLM 5.3 and DeepSeek V4.1 Flash take 1,048,576 tokens of context; Gemma’s 32,768 fills fast. Prices for every model.
How we tested this
On 2026-11-15 we installed haystack-ai 3.5.0 in a clean virtual environment and ran the plain, streaming, tool and Agent examples above against a local mock server that speaks BroadNet’s chat-completions shape. We read the logged requests. All went to /v1/chat/completions with the model ID unchanged, a Bearer key, max_tokens from generation_kwargs and, for tools, the function schema. Haystack also sends n: 1. We had no BroadNet key, so we did not test real model output.
Troubleshooting
- 401. The key is wrong, expired or out of allowance, or
BROADNET_API_KEYisn’t set in the process that runs the pipeline. - 404.
api_base_urlmust end in/v1, and the model ID must match BroadNet’s exactly. Paid models need a top-up; the $5 credit covers onlybroadnet/gemma-4-26b-a4band the two Transcribe models. - Empty replies. DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason before answering, and the reasoning counts toward
max_tokens. Raise it. Why answers come back empty. - 429 in batch indexing or evaluation. A pay-as-you-go key allows 30 requests a minute and 4 in flight. Use the generator’s
max_retriesand see errors and retries.
