BroadRouter
Get started

Use Haystack pipelines and agents with BroadNet Inference

Updated

Install haystack-ai and create OpenAIChatGenerator with api_base_url set to https://inference.broadnet.ai/v1, api_key set to a Secret read from BROADNET_API_KEY, and model set to a BroadNet ID such as broadnet/glm-5.3. Use it in pipelines or a Haystack Agent. Pass max_tokens in generation_kwargs, and use GLM 5.3 or Qwen3.8 27B when the generator has tools.

Haystack is deepset’s open-source framework for RAG pipelines and agents. Its OpenAIChatGenerator talks to any OpenAI-compatible server through api_base_url. BroadNet Inference serves standard chat completions at https://inference.broadnet.ai/v1, so that is the component to use. BroadRouter is BroadNet’s own site; this follows the Haystack 3.5 docs.

Install

pip install haystack-ai
export BROADNET_API_KEY='your-broadnet-key'

Connect

from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret

generator = OpenAIChatGenerator(
    api_base_url="https://inference.broadnet.ai/v1",
    api_key=Secret.from_env_var("BROADNET_API_KEY"),
    model="broadnet/glm-5.3",
    generation_kwargs={"max_tokens": 4000},
)

reply = generator.run([ChatMessage.from_user("Name three uses of a RAG pipeline.")])
print(reply["replies"][0].text)
  • model is sent unchanged. Use the BroadNet ID with its broadnet/ prefix.
  • api_key is a Haystack Secret. Without it, Haystack looks for OPENAI_API_KEY.
  • generation_kwargs are passed into the chat completion request. Set max_tokens high enough for reasoning (see below).

Stream the answer

from haystack.components.generators.utils import print_streaming_chunk

generator = OpenAIChatGenerator(
    api_base_url="https://inference.broadnet.ai/v1",
    api_key=Secret.from_env_var("BROADNET_API_KEY"),
    model="broadnet/glm-5.3",
    streaming_callback=print_streaming_chunk,
)

Haystack then sends stream: true with stream_options.include_usage, so token counts arrive in the last chunk. BroadNet’s docs describe the same option.

Tools and agents

from haystack.components.agents import Agent
from haystack.tools import Tool

def get_weather(city: str) -> str:
    return f"21C in {city}"

weather = Tool(
    name="get_weather",
    description="Get the weather for a city",
    parameters={"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]},
    function=get_weather,
)

chat = OpenAIChatGenerator(
    api_base_url="https://inference.broadnet.ai/v1",
    api_key=Secret.from_env_var("BROADNET_API_KEY"),
    model="broadnet/glm-5.3",
    tools=[weather],
)
agent = Agent(chat_generator=chat, tools=[weather])
result = agent.run(messages=[ChatMessage.from_user("What's the weather in Riyadh?")])
print(result["messages"][-1].text)

BroadNet’s docs list tools and structured output for broadnet/glm-5.3 and broadnet/qwen3.8-27b only. Keep tools_strict at its default of False unless you have tested strict schemas.

Which model to pick

Pipeline Model
Agents and tool calls broadnet/glm-5.3
RAG over scanned pages or Arabic text, with tools broadnet/qwen3.8-27b
Long-context answering on a flat-price plan broadnet/deepseek-v4.1-flash
Trying it out on the free starting credit broadnet/gemma-4-26b-a4b (32,768-token context)

Retrieved context can be large. GLM 5.3 and DeepSeek V4.1 Flash take 1,048,576 tokens of context; Gemma’s 32,768 fills fast. Prices for every model.

How we tested this

On 2026-11-15 we installed haystack-ai 3.5.0 in a clean virtual environment and ran the plain, streaming, tool and Agent examples above against a local mock server that speaks BroadNet’s chat-completions shape. We read the logged requests. All went to /v1/chat/completions with the model ID unchanged, a Bearer key, max_tokens from generation_kwargs and, for tools, the function schema. Haystack also sends n: 1. We had no BroadNet key, so we did not test real model output.

Troubleshooting

  • 401. The key is wrong, expired or out of allowance, or BROADNET_API_KEY isn’t set in the process that runs the pipeline.
  • 404. api_base_url must end in /v1, and the model ID must match BroadNet’s exactly. Paid models need a top-up; the $5 credit covers only broadnet/gemma-4-26b-a4b and the two Transcribe models.
  • Empty replies. DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason before answering, and the reasoning counts toward max_tokens. Raise it. Why answers come back empty.
  • 429 in batch indexing or evaluation. A pay-as-you-go key allows 30 requests a minute and 4 in flight. Use the generator’s max_retries and see errors and retries.

Questions

Which Haystack parameter sets the BroadNet URL?

api_base_url on OpenAIChatGenerator. Haystack's docs describe it as the way to reach custom deployments. Set it to https://inference.broadnet.ai/v1.

Should I use OpenAIResponsesChatGenerator with BroadNet?

No. That component calls OpenAI's Responses API, which BroadNet doesn't list. Use OpenAIChatGenerator, which calls chat completions.

Why does Haystack fail with a missing OPENAI_API_KEY?

The api_key default is Secret.from_env_var("OPENAI_API_KEY"). Pass api_key=Secret.from_env_var("BROADNET_API_KEY") so Haystack reads your BroadNet key instead.

Can a Haystack Agent call tools on BroadNet?

Yes, with a model BroadNet lists with tools, GLM 5.3 or Qwen3.8 27B. In our mock-server test, the Agent sent the tool schema over chat completions and got the tool call back.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity