BroadRouter
Get started

LLM streaming explained: SSE chunks, deltas, usage and cancel

By the BroadNet Inference team · 4 min read · Published · Updated

An LLM API streams a chat completion as Server-Sent Events: each data line holds a JSON chunk whose choices[0].delta carries the next piece of text. Concatenate the deltas, stop at data: [DONE], and set stream_options include_usage to get token counts in a final chunk with an empty choices list. To cancel, close the connection.

Without streaming, a chat completion returns nothing until the whole answer is ready. A 600-token reply at about 100 tokens per second means several seconds of silence. With streaming, the first words show up as soon as the model writes them. This post covers what actually travels over the wire, how to parse it, how to get token usage out of a stream, and how to stop one cleanly. For the BroadNet-specific rules in short form, see the streaming reference guide.

What does an SSE stream look like on the wire?

Server-Sent Events is a plain HTTP response with Content-Type: text/event-stream. The server keeps the connection open and writes events. Each event is one or more data: lines followed by a blank line. OpenAI-compatible APIs put one JSON chunk in each event and end with a literal [DONE]:

data: {"id":"c1","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}],"usage":null}

data: {"id":"c1","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"One, "},"finish_reason":null}],"usage":null}

data: {"id":"c1","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"two, three."},"finish_reason":"stop"}],"usage":null}

data: {"id":"c1","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":14,"completion_tokens":6,"total_tokens":20}}

data: [DONE]

The shape above is illustrative and trimmed. Real chunks also carry model and created, and reasoning models may report reasoning tokens in usage.

What is in a chunk?

Field Where What it means
choices[0].delta.role First chunk Usually assistant. Sent once.
choices[0].delta.content Most chunks The next piece of text. Can be empty.
choices[0].delta.tool_calls Tool calls Fragments of the function name and JSON arguments, to join by index.
choices[0].finish_reason Last content chunk stop, length or tool_calls. null until then.
usage Final chunk, if requested Totals for the whole request. null everywhere else.

The key idea is delta, not snapshot. Each chunk holds only what is new. You rebuild the message by joining the pieces in order. Tool calls stream the same way: the arguments arrive as string fragments that are not valid JSON until the last one lands, so parse them only after finish_reason is tool_calls.

How do you get token usage from a stream?

A non-streamed response always includes usage. A stream does not, unless you ask. Set stream_options: {"include_usage": true} and the server sends one extra chunk before [DONE]. That chunk has the totals and an empty choices array. On BroadNet Inference the usage can also include prompt_tokens_details.cached_tokens and completion_tokens_details.reasoning_tokens when the model reports them, which is what you need for cost logging.

The common bug: code that reads chunk.choices[0] on every chunk throws an index error on the usage chunk. Check choices first.

Python: stream, collect usage, cancel

import os
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])

stream = client.chat.completions.create(
    model="broadnet/deepseek-v4.1-flash",  # DeepSeek V4.1 Flash, a reasoning model
    messages=[{"role": "user", "content": "Explain SSE in three sentences."}],
    stream=True,
    stream_options={"include_usage": True},
    max_tokens=2000,  # reasoning counts toward this limit
)

def user_pressed_stop():  # replace with your own condition
    return False

parts, usage, finish = [], None, None
try:
    for chunk in stream:
        if chunk.choices:
            choice = chunk.choices[0]
            if choice.delta.content:
                parts.append(choice.delta.content)
                print(choice.delta.content, end="", flush=True)
            if choice.finish_reason:
                finish = choice.finish_reason
        if chunk.usage:
            usage = chunk.usage
        if user_pressed_stop():  # your own condition
            break
finally:
    stream.close()  # closes the HTTP connection

text = "".join(parts)

break alone stops reading, but the socket stays open until it is garbage collected. Close it explicitly so the server sees the disconnect.

Node: stream with an AbortController

import OpenAI from "openai";

const client = new OpenAI({ baseURL: "https://inference.broadnet.ai/v1", apiKey: process.env.BROADNET_API_KEY });
const controller = new AbortController();
setTimeout(() => controller.abort(), 30_000); // hard stop after 30 s

const stream = await client.chat.completions.create(
  {
    model: "broadnet/deepseek-v4.1-flash",
    messages: [{ role: "user", content: "Explain SSE in three sentences." }],
    stream: true,
    stream_options: { include_usage: true },
  },
  { signal: controller.signal },
);

let text = "";
for await (const chunk of stream) {
  const delta = chunk.choices[0]?.delta?.content;
  if (delta) text += delta;
  if (chunk.usage) console.log(chunk.usage);
}

The openai-node README documents the signal option, and it works while the body is still being read. In a web server, wire controller.abort() to the client request’s close event so a user who leaves the page stops the generation too. More setup in the Node SDK guide.

curl: see the raw events

curl -N https://inference.broadnet.ai/v1/chat/completions \
  -H "Authorization: Bearer $BROADNET_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"broadnet/gemma-4-26b-a4b","stream":true,"stream_options":{"include_usage":true},"messages":[{"role":"user","content":"Count to five."}]}'

-N turns off output buffering so events print as they arrive. Press Ctrl+C to cancel; that closes the connection. The API request builder generates this request, and the Python and Node versions, for any model and base URL.

How do you cancel a stream, and what does it cost?

Cancellation in SSE is just closing the connection. There is no “stop” message. What happens next is up to the server: a well-behaved one notices the closed socket and stops generating. Tokens produced before that moment were real work. BroadNet’s API reference does not document billing for cancelled streams, so plan as if generated tokens up to the disconnect are billed on pay-as-you-go, and check your usage records. On a flat plan, a cancelled stream costs nothing extra but frees the session only when the server stops.

Five streaming bugs to check for

  1. Buffering proxies. Nginx and some CDNs buffer responses, so the client gets everything at once. Disable proxy buffering for the streaming route.
  2. No [DONE] check. A dropped connection looks like a short answer. Treat a missing [DONE] as incomplete and record the X-Request-ID, as the errors guide describes.
  3. Retrying mid-stream. SDKs do not retry once the body starts arriving. A retry is a new request and, on pay-as-you-go, a new bill.
  4. Ignoring finish_reason: length. On reasoning models such as DeepSeek V4.1 Flash the thinking counts toward max_tokens, so a small limit can stop the stream before any visible text. What reasoning tokens cost.
  5. Splitting lines wrong. If you parse SSE by hand, buffer bytes until you see a blank line. A TCP read can end in the middle of a JSON chunk.

Where BroadNet Inference fits

BroadNet Inference uses the standard OpenAI-compatible streaming shape, so the Python and Node SDKs parse it unchanged. Usage in the final chunk includes cached and reasoning tokens when the model reports them. Its docs say streaming should still be verified per model before production use, so run the code above against the exact model you ship. If you mostly care about how fast the first token arrives, read TTFT vs throughput next. If you need realtime voice over WebSockets, this API is not built for that; it offers chat completions and file transcription.

Questions

What is the difference between SSE and WebSockets for LLM streaming?

SSE is one-way, server to client, over a normal HTTP response. That is all a chat completion needs, so OpenAI-compatible APIs use SSE. WebSockets are two-way and need a protocol upgrade; they are used for realtime voice, not for chat completions.

Why is usage null in my streaming chunks?

Usage is only sent when you ask for it. Add stream_options with include_usage set to true. Every chunk except the last then has usage null, and the last chunk carries the totals with an empty choices list.

Am I billed if I cancel a stream?

Providers differ and many do not document it. Assume tokens generated before the server noticed the closed connection are billed. Log the X-Request-ID and compare against your usage records.

How do I know a stream finished normally?

You received data: [DONE] and the last content chunk had a finish_reason such as stop. A finish_reason of length means max_tokens cut the answer. No [DONE] means the connection dropped.

Does streaming make the model faster?

No. Total generation time is the same. Streaming lowers the time until the user sees the first words, which is what makes a chat interface feel fast.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity