Streaming lets an agent or a chat interface show the answer as it is generated. BroadNet Inference uses the standard chat-completions streaming shape, so the OpenAI Python SDK and OpenAI Node SDK handle the parsing for you.
With curl
curl -N -D stream-headers.txt https://inference.broadnet.ai/v1/chat/completions \
-H "Authorization: Bearer $BROADNET_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"broadnet/gemma-4-26b-a4b","stream":true,"stream_options":{"include_usage":true},"messages":[{"role":"user","content":"Count from one to three."}]}'
-N turns off curl’s buffering, and -D saves the headers, including X-Request-ID.
What does the stream look like?
The response is a Server-Sent Events stream. Under the WHATWG HTML standard, each event is one or more data: lines followed by a blank line, and a line that starts with a colon is a comment you can ignore. Each data: line holds one JSON chunk, and the text arrives in choices[0].delta.content.
The rules
- Read events until the literal
data: [DONE]line, then close the reader. - With
stream_options: {"include_usage": true}, one extra chunk arrives just before[DONE]. It carriesusagefor the whole request and an emptychoiceslist, so checkchoicesbefore you index into it. - When the model reports them, that usage includes
completion_tokens_details.reasoning_tokens(DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason first) andprompt_tokens_details.cached_tokens. Cached tokens are billed at the lower cached-input price.
With the Python SDK
import os
from openai import OpenAI
client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
stream = client.chat.completions.create(
model="broadnet/gemma-4-26b-a4b",
messages=[{"role": "user", "content": "Count from one to three."}],
stream=True,
stream_options={"include_usage": True},
)
usage = None
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
if chunk.usage:
usage = chunk.usage
print()
print(usage) # None means the stream ended before the usage chunk
The SDK reads up to [DONE] for you. The OpenAI Node SDK guide shows the same loop in JavaScript.
When the stream ends early
If the connection closes before [DONE], the answer is incomplete. BroadNet’s guidance:
- record the
X-Request-ID; - decide whether your application can safely make a new request. A retry is a new inference and is billed again on pay-as-you-go.
For agents on an unlimited plan, retries cost nothing extra, but they still occupy the session while they run.
BroadNet’s docs say per-model release verification is still required before you rely on streaming in production, so test it on the model you ship with.
Troubleshooting
- Everything prints at the end. curl needs
-N. Behind your own proxy, turn off response buffering fortext/event-stream. IndexErrororundefinedon the last chunk. The usage chunk has an emptychoiceslist. Checkchunk.choicesbefore reading[0].- No usage at all. You did not send
stream_options, or the stream broke before the final chunk. - Stream stops with
finish_reasonlengthand little text. A reasoning model spentmax_tokenson reasoning. See why reasoning models return empty answers. - 429 while streaming in parallel. Each open stream counts as a request in flight; a pay-as-you-go key allows 4. See BroadNet error codes and retries.
