---
title: "Haystack OpenAIChatGenerator with a custom api_base_url"
description: "Point Haystack's OpenAIChatGenerator at BroadNet Inference with api_base_url, a BroadNet model ID and a Secret key, for pipelines, tools, streaming and agents."
url: https://broadrouter.com/guides/haystack-openai-compatible/
lang: en
updated: 2026-11-16
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Use Haystack pipelines and agents with BroadNet Inference

Updated 2026-11-16

Install haystack-ai and create OpenAIChatGenerator with api\_base\_url set to https://inference.broadnet.ai/v1, api\_key set to a Secret read from BROADNET\_API\_KEY, and model set to a BroadNet ID such as broadnet/glm-5.3. Use it in pipelines or a Haystack Agent. Pass max\_tokens in generation\_kwargs, and use GLM 5.3 or Qwen3.8 27B when the generator has tools.

Haystack is deepset’s open-source framework for RAG pipelines and agents. Its `OpenAIChatGenerator` talks to any OpenAI-compatible server through `api_base_url`. BroadNet Inference serves standard chat completions at `https://inference.broadnet.ai/v1`, so that is the component to use. BroadRouter is BroadNet’s own site; this follows the Haystack 3.5 docs.

## Install

```bash
pip install haystack-ai
export BROADNET_API_KEY='your-broadnet-key'
```

## Connect

```python
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret

generator = OpenAIChatGenerator(
    api_base_url="https://inference.broadnet.ai/v1",
    api_key=Secret.from_env_var("BROADNET_API_KEY"),
    model="broadnet/glm-5.3",
    generation_kwargs={"max_tokens": 4000},
)

reply = generator.run([ChatMessage.from_user("Name three uses of a RAG pipeline.")])
print(reply["replies"][0].text)
```

-   **`model`** is sent unchanged. Use the BroadNet ID with its `broadnet/` prefix.
-   **`api_key`** is a Haystack `Secret`. Without it, Haystack looks for `OPENAI_API_KEY`.
-   **`generation_kwargs`** are passed into the chat completion request. Set `max_tokens` high enough for reasoning (see below).

## Stream the answer

```python
from haystack.components.generators.utils import print_streaming_chunk

generator = OpenAIChatGenerator(
    api_base_url="https://inference.broadnet.ai/v1",
    api_key=Secret.from_env_var("BROADNET_API_KEY"),
    model="broadnet/glm-5.3",
    streaming_callback=print_streaming_chunk,
)
```

Haystack then sends `stream: true` with `stream_options.include_usage`, so token counts arrive in the last chunk. BroadNet’s docs describe the same option.

## Tools and agents

```python
from haystack.components.agents import Agent
from haystack.tools import Tool

def get_weather(city: str) -> str:
    return f"21C in {city}"

weather = Tool(
    name="get_weather",
    description="Get the weather for a city",
    parameters={"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]},
    function=get_weather,
)

chat = OpenAIChatGenerator(
    api_base_url="https://inference.broadnet.ai/v1",
    api_key=Secret.from_env_var("BROADNET_API_KEY"),
    model="broadnet/glm-5.3",
    tools=[weather],
)
agent = Agent(chat_generator=chat, tools=[weather])
result = agent.run(messages=[ChatMessage.from_user("What's the weather in Riyadh?")])
print(result["messages"][-1].text)
```

BroadNet’s docs list tools and structured output for `broadnet/glm-5.3` and `broadnet/qwen3.8-27b` only. Keep `tools_strict` at its default of `False` unless you have tested strict schemas.

## Which model to pick

| Pipeline | Model |
| --- | --- |
| Agents and tool calls | `broadnet/glm-5.3` |
| RAG over scanned pages or Arabic text, with tools | `broadnet/qwen3.8-27b` |
| Long-context answering on a flat-price plan | `broadnet/deepseek-v4.1-flash` |
| Trying it out on the free starting credit | `broadnet/gemma-4-26b-a4b` (32,768-token context) |

Retrieved context can be large. GLM 5.3 and DeepSeek V4.1 Flash take 1,048,576 tokens of context; Gemma’s 32,768 fills fast. [Prices for every model](https://broadrouter.com/pricing/).

## How we tested this

On 2026-11-15 we installed haystack-ai 3.5.0 in a clean virtual environment and ran the plain, streaming, tool and Agent examples above against a local mock server that speaks BroadNet’s chat-completions shape. We read the logged requests. All went to `/v1/chat/completions` with the model ID unchanged, a Bearer key, `max_tokens` from `generation_kwargs` and, for tools, the function schema. Haystack also sends `n: 1`. We had no BroadNet key, so we did not test real model output.

## Troubleshooting

-   **401.** The key is wrong, expired or out of allowance, or `BROADNET_API_KEY` isn’t set in the process that runs the pipeline.
-   **404.** `api_base_url` must end in `/v1`, and the model ID must match BroadNet’s exactly. Paid models need a top-up; the $5 credit covers only `broadnet/gemma-4-26b-a4b` and the two Transcribe models.
-   **Empty replies.** DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason before answering, and the reasoning counts toward `max_tokens`. Raise it. [Why answers come back empty](https://broadrouter.com/guides/reasoning-tokens-empty-response/).
-   **429 in batch indexing or evaluation.** A pay-as-you-go key allows 30 requests a minute and 4 in flight. Use the generator’s `max_retries` and see [errors and retries](https://broadrouter.com/guides/errors-and-retries/).

## Terms in this guide

[Tool calling](https://broadrouter.com/glossary/tool-calling/)

Tool calling, also called function calling, is an LLM API feature in which the model returns a structured request to run a function the developer has defined, with JSON arguments. The application runs the function and sends the result back; the model never runs code itself.

[Pay-as-you-go](https://broadrouter.com/glossary/pay-as-you-go/)

Pay-as-you-go is a pricing model in which you pay only for what you use, metered per unit such as tokens, requests or audio minutes, with no subscription or minimum commitment. Most LLM APIs bill this way, from a prepaid balance or a periodic invoice.

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[Chat Completions API](https://broadrouter.com/glossary/chat-completions-api/)

The Chat Completions API is OpenAI's HTTP endpoint, POST /v1/chat/completions, that takes a model name and a list of role-tagged messages and returns the model's next message. Its request and response shape became the common format that OpenAI-compatible providers and SDKs reuse.

[Concurrency](https://broadrouter.com/glossary/concurrency/)

Concurrency is the number of requests a client has in flight at the same time. LLM APIs often cap it per key or plan, separately from requests per minute, because each open request holds GPU capacity until its response has finished.

[Prepaid credit](https://broadrouter.com/glossary/prepaid-credit/)

Prepaid credit is money added to an API account before use. Each request deducts its cost from the balance, and when the balance runs out, metered requests stop until you top up. It caps spend at what you have paid in.

[RAG (retrieval-augmented generation)](https://broadrouter.com/glossary/rag/)

RAG (retrieval-augmented generation) is a pattern in which an application first retrieves passages relevant to a question from an external store, such as a search or vector index, and then adds them to the prompt so a language model can answer from that text rather than from memory alone.

[Rate limit](https://broadrouter.com/glossary/rate-limit/)

A rate limit is a cap an API places on how much a client can use it in a time window, such as requests per minute or tokens per minute. Requests over the limit are refused, usually with HTTP 429 Too Many Requests, until usage falls back under it.

## Questions

### Which Haystack parameter sets the BroadNet URL?

api\_base\_url on OpenAIChatGenerator. Haystack's docs describe it as the way to reach custom deployments. Set it to https://inference.broadnet.ai/v1.

### Should I use OpenAIResponsesChatGenerator with BroadNet?

No. That component calls OpenAI's Responses API, which BroadNet doesn't list. Use OpenAIChatGenerator, which calls chat completions.

### Why does Haystack fail with a missing OPENAI\_API\_KEY?

The api\_key default is Secret.from\_env\_var("OPENAI\_API\_KEY"). Pass api\_key=Secret.from\_env\_var("BROADNET\_API\_KEY") so Haystack reads your BroadNet key instead.

### Can a Haystack Agent call tools on BroadNet?

Yes, with a model BroadNet lists with tools, GLM 5.3 or Qwen3.8 27B. In our mock-server test, the Agent sent the tool schema over chat completions and got the tool call back.

## Related guides

-   [Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/)
-   [Use LlamaIndex with BroadNet Inference through OpenAILike](https://broadrouter.com/guides/llamaindex-openailike/)
-   [Why a reasoning model returns an empty answer](https://broadrouter.com/guides/reasoning-tokens-empty-response/)
-   [BroadNet Inference error codes and when to retry](https://broadrouter.com/guides/errors-and-retries/)
-   [All tools that work with BroadNet Inference](https://broadrouter.com/integrations/): SDKs, coding agents, chat UIs and automation tools, with exact settings for each

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=guides-haystack-openai-compatible&utm_content=page-cta) ·

**Sources**

-   [Haystack docs 3.5, OpenAIChatGenerator (checked 2026-11-15)](https://docs.haystack.deepset.ai/docs/openaichatgenerator)
-   [PyPI, haystack-ai 3.5.0, Python 3.11+ (checked 2026-11-15)](https://pypi.org/project/haystack-ai/)
-   [BroadNet Inference API reference (checked 2026-11-15)](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=guides-haystack-openai-compatible&utm_content=sources)
