Without streaming, a chat completion returns nothing until the whole answer is ready. A 600-token reply at about 100 tokens per second means several seconds of silence. With streaming, the first words show up as soon as the model writes them. This post covers what actually travels over the wire, how to parse it, how to get token usage out of a stream, and how to stop one cleanly. For the BroadNet-specific rules in short form, see the streaming reference guide.
What does an SSE stream look like on the wire?
Server-Sent Events is a plain HTTP response with Content-Type: text/event-stream. The server keeps the connection open and writes events. Each event is one or more data: lines followed by a blank line. OpenAI-compatible APIs put one JSON chunk in each event and end with a literal [DONE]:
data: {"id":"c1","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}],"usage":null}
data: {"id":"c1","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"One, "},"finish_reason":null}],"usage":null}
data: {"id":"c1","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"two, three."},"finish_reason":"stop"}],"usage":null}
data: {"id":"c1","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":14,"completion_tokens":6,"total_tokens":20}}
data: [DONE]
The shape above is illustrative and trimmed. Real chunks also carry model and created, and reasoning models may report reasoning tokens in usage.
What is in a chunk?
| Field | Where | What it means |
|---|---|---|
choices[0].delta.role |
First chunk | Usually assistant. Sent once. |
choices[0].delta.content |
Most chunks | The next piece of text. Can be empty. |
choices[0].delta.tool_calls |
Tool calls | Fragments of the function name and JSON arguments, to join by index. |
choices[0].finish_reason |
Last content chunk | stop, length or tool_calls. null until then. |
usage |
Final chunk, if requested | Totals for the whole request. null everywhere else. |
The key idea is delta, not snapshot. Each chunk holds only what is new. You rebuild the message by joining the pieces in order. Tool calls stream the same way: the arguments arrive as string fragments that are not valid JSON until the last one lands, so parse them only after finish_reason is tool_calls.
How do you get token usage from a stream?
A non-streamed response always includes usage. A stream does not, unless you ask. Set stream_options: {"include_usage": true} and the server sends one extra chunk before [DONE]. That chunk has the totals and an empty choices array. On BroadNet Inference the usage can also include prompt_tokens_details.cached_tokens and completion_tokens_details.reasoning_tokens when the model reports them, which is what you need for cost logging.
The common bug: code that reads chunk.choices[0] on every chunk throws an index error on the usage chunk. Check choices first.
Python: stream, collect usage, cancel
import os
from openai import OpenAI
client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
stream = client.chat.completions.create(
model="broadnet/deepseek-v4.1-flash", # DeepSeek V4.1 Flash, a reasoning model
messages=[{"role": "user", "content": "Explain SSE in three sentences."}],
stream=True,
stream_options={"include_usage": True},
max_tokens=2000, # reasoning counts toward this limit
)
def user_pressed_stop(): # replace with your own condition
return False
parts, usage, finish = [], None, None
try:
for chunk in stream:
if chunk.choices:
choice = chunk.choices[0]
if choice.delta.content:
parts.append(choice.delta.content)
print(choice.delta.content, end="", flush=True)
if choice.finish_reason:
finish = choice.finish_reason
if chunk.usage:
usage = chunk.usage
if user_pressed_stop(): # your own condition
break
finally:
stream.close() # closes the HTTP connection
text = "".join(parts)
break alone stops reading, but the socket stays open until it is garbage collected. Close it explicitly so the server sees the disconnect.
Node: stream with an AbortController
import OpenAI from "openai";
const client = new OpenAI({ baseURL: "https://inference.broadnet.ai/v1", apiKey: process.env.BROADNET_API_KEY });
const controller = new AbortController();
setTimeout(() => controller.abort(), 30_000); // hard stop after 30 s
const stream = await client.chat.completions.create(
{
model: "broadnet/deepseek-v4.1-flash",
messages: [{ role: "user", content: "Explain SSE in three sentences." }],
stream: true,
stream_options: { include_usage: true },
},
{ signal: controller.signal },
);
let text = "";
for await (const chunk of stream) {
const delta = chunk.choices[0]?.delta?.content;
if (delta) text += delta;
if (chunk.usage) console.log(chunk.usage);
}
The openai-node README documents the signal option, and it works while the body is still being read. In a web server, wire controller.abort() to the client request’s close event so a user who leaves the page stops the generation too. More setup in the Node SDK guide.
curl: see the raw events
curl -N https://inference.broadnet.ai/v1/chat/completions \
-H "Authorization: Bearer $BROADNET_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"broadnet/gemma-4-26b-a4b","stream":true,"stream_options":{"include_usage":true},"messages":[{"role":"user","content":"Count to five."}]}'
-N turns off output buffering so events print as they arrive. Press Ctrl+C to cancel; that closes the connection. The API request builder generates this request, and the Python and Node versions, for any model and base URL.
How do you cancel a stream, and what does it cost?
Cancellation in SSE is just closing the connection. There is no “stop” message. What happens next is up to the server: a well-behaved one notices the closed socket and stops generating. Tokens produced before that moment were real work. BroadNet’s API reference does not document billing for cancelled streams, so plan as if generated tokens up to the disconnect are billed on pay-as-you-go, and check your usage records. On a flat plan, a cancelled stream costs nothing extra but frees the session only when the server stops.
Five streaming bugs to check for
- Buffering proxies. Nginx and some CDNs buffer responses, so the client gets everything at once. Disable proxy buffering for the streaming route.
- No
[DONE]check. A dropped connection looks like a short answer. Treat a missing[DONE]as incomplete and record theX-Request-ID, as the errors guide describes. - Retrying mid-stream. SDKs do not retry once the body starts arriving. A retry is a new request and, on pay-as-you-go, a new bill.
- Ignoring
finish_reason: length. On reasoning models such as DeepSeek V4.1 Flash the thinking counts toward max_tokens, so a small limit can stop the stream before any visible text. What reasoning tokens cost. - Splitting lines wrong. If you parse SSE by hand, buffer bytes until you see a blank line. A TCP read can end in the middle of a JSON chunk.
Where BroadNet Inference fits
BroadNet Inference uses the standard OpenAI-compatible streaming shape, so the Python and Node SDKs parse it unchanged. Usage in the final chunk includes cached and reasoning tokens when the model reports them. Its docs say streaming should still be verified per model before production use, so run the code above against the exact model you ship. If you mostly care about how fast the first token arrives, read TTFT vs throughput next. If you need realtime voice over WebSockets, this API is not built for that; it offers chat completions and file transcription.
