BroadRouter
Get started

Streaming

Updated

Streaming is a way of returning a language model response in small chunks while it is still being generated, instead of as one reply at the end. OpenAI-compatible APIs stream over server-sent events when the request sets stream to true.

Also called: streaming responses, stream=true, token streaming

Without streaming, the client waits until the whole answer is finished. With it, text appears as the model writes it.

How it works

Streaming uses server-sent events (SSE). The server answers with content type text/event-stream and keeps the connection open. MDN describes the format: lines of field: value, with messages “separated by a pair of newline characters.”

In the Chat Completions format, each event carries a chunk with a small delta:

data: {"choices":[{"index":0,"delta":{"content":"Hel"}}]}

data: {"choices":[{"index":0,"delta":{"content":"lo"}}]}

data: {"choices":[],"usage":{"prompt_tokens":12,"completion_tokens":2}}

data: [DONE]

The client joins the deltas. The literal data: [DONE] line marks the end. The usage chunk appears only if the request asked for it with stream_options: {"include_usage": true}.

Why it matters

  • Perceived speed. Users see the first words after the time to first token, not after the whole reply. After that, text flows at the model’s tokens per second.
  • Long outputs. Streaming keeps the connection active, which helps with proxies that close idle requests.
  • Early action. Agents and UIs can start parsing or rendering before the end.

There are costs. OpenAI notes that streaming makes moderation harder, because “partial completions may be more difficult to evaluate.” A stream can also break halfway, and a streamed reply holds a concurrency slot until it ends.

On BroadNet Inference

BroadNet uses the standard chat-completions streaming shape on https://inference.broadnet.ai/v1/chat/completions, so the Python and Node SDKs parse it for you. With curl, add -N to turn off buffering:

curl -N https://inference.broadnet.ai/v1/chat/completions \
  -H "Authorization: Bearer $BROADNET_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"broadnet/gemma-4-26b-a4b","stream":true,"stream_options":{"include_usage":true},"messages":[{"role":"user","content":"Count to three."}]}'

Reasoning models can report reasoning_tokens and cached_tokens in the final usage chunk. If the connection closes before [DONE], treat the answer as incomplete, record the X-Request-ID header, and decide whether a new request is safe. BroadNet’s docs say to verify streaming on the model you ship with before relying on it in production. Details are in the streaming guide and errors and retries.

Common mistakes

  • Treating a closed connection as a finished answer. Only [DONE] means complete.
  • Putting a buffering proxy in front of the client. The reply then arrives all at once.

Questions

Does streaming make the model faster?

No. Generation speed is the same. Streaming shows the first words sooner, so the reply feels faster, and lets code act on partial output.

How do I get token usage in a stream?

In the Chat Completions format, send stream_options with include_usage set to true. The last chunk then carries usage and an empty choices list.

Is a retried stream billed twice?

On a per-token key, usually yes. A retry is a new inference, billed again. On a flat plan with no token meter, it only costs time.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity