Without streaming, the client waits until the whole answer is finished. With it, text appears as the model writes it.
How it works
Streaming uses server-sent events (SSE). The server answers with content type text/event-stream and keeps the connection open. MDN describes the format: lines of field: value, with messages “separated by a pair of newline characters.”
In the Chat Completions format, each event carries a chunk with a small delta:
data: {"choices":[{"index":0,"delta":{"content":"Hel"}}]}
data: {"choices":[{"index":0,"delta":{"content":"lo"}}]}
data: {"choices":[],"usage":{"prompt_tokens":12,"completion_tokens":2}}
data: [DONE]
The client joins the deltas. The literal data: [DONE] line marks the end. The usage chunk appears only if the request asked for it with stream_options: {"include_usage": true}.
Why it matters
- Perceived speed. Users see the first words after the time to first token, not after the whole reply. After that, text flows at the model’s tokens per second.
- Long outputs. Streaming keeps the connection active, which helps with proxies that close idle requests.
- Early action. Agents and UIs can start parsing or rendering before the end.
There are costs. OpenAI notes that streaming makes moderation harder, because “partial completions may be more difficult to evaluate.” A stream can also break halfway, and a streamed reply holds a concurrency slot until it ends.
On BroadNet Inference
BroadNet uses the standard chat-completions streaming shape on https://inference.broadnet.ai/v1/chat/completions, so the Python and Node SDKs parse it for you. With curl, add -N to turn off buffering:
curl -N https://inference.broadnet.ai/v1/chat/completions \
-H "Authorization: Bearer $BROADNET_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"broadnet/gemma-4-26b-a4b","stream":true,"stream_options":{"include_usage":true},"messages":[{"role":"user","content":"Count to three."}]}'
Reasoning models can report reasoning_tokens and cached_tokens in the final usage chunk. If the connection closes before [DONE], treat the answer as incomplete, record the X-Request-ID header, and decide whether a new request is safe. BroadNet’s docs say to verify streaming on the model you ship with before relying on it in production. Details are in the streaming guide and errors and retries.
Common mistakes
- Treating a closed connection as a finished answer. Only
[DONE]means complete. - Putting a buffering proxy in front of the client. The reply then arrives all at once.