---
title: "Stream LLM responses over SSE, with usage and retries"
description: "How streaming works on BroadNet Inference - Server-Sent Events, the [DONE] marker, token usage in the last chunk, and what to do when a stream ends early."
url: https://broadrouter.com/guides/streaming-chat-completions/
lang: en
updated: 2026-11-15
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Stream chat completions and handle dropped connections

Updated 2026-11-15

Set stream to true on /v1/chat/completions and read Server-Sent Events until the literal data: \[DONE\] marker. Add stream\_options include\_usage to get token counts in the last chunk. If the connection ends before \[DONE\], treat the answer as incomplete and log the X-Request-ID.

Streaming lets an agent or a chat interface show the answer as it is generated. BroadNet Inference uses the standard chat-completions streaming shape, so the [OpenAI Python SDK](https://broadrouter.com/guides/openai-python-sdk/) and [OpenAI Node SDK](https://broadrouter.com/guides/openai-node-sdk/) handle the parsing for you.

## With curl

```bash
curl -N -D stream-headers.txt https://inference.broadnet.ai/v1/chat/completions \
  -H "Authorization: Bearer $BROADNET_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"broadnet/gemma-4-26b-a4b","stream":true,"stream_options":{"include_usage":true},"messages":[{"role":"user","content":"Count from one to three."}]}'
```

`-N` turns off curl’s buffering, and `-D` saves the headers, including `X-Request-ID`.

## What does the stream look like?

The response is a [Server-Sent Events](https://broadrouter.com/glossary/sse/) stream. Under the WHATWG HTML standard, each event is one or more `data:` lines followed by a blank line, and a line that starts with a colon is a comment you can ignore. Each `data:` line holds one JSON chunk, and the text arrives in `choices[0].delta.content`.

## The rules

1.  Read events until the literal `data: [DONE]` line, then close the reader.
2.  With `stream_options: {"include_usage": true}`, one extra chunk arrives just before `[DONE]`. It carries `usage` for the whole request and an empty `choices` list, so check `choices` before you index into it.
3.  When the model reports them, that usage includes `completion_tokens_details.reasoning_tokens` (DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason first) and `prompt_tokens_details.cached_tokens`. Cached tokens are billed at the lower cached-input price.

## With the Python SDK

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])

stream = client.chat.completions.create(
    model="broadnet/gemma-4-26b-a4b",
    messages=[{"role": "user", "content": "Count from one to three."}],
    stream=True,
    stream_options={"include_usage": True},
)
usage = None
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:
        usage = chunk.usage
print()
print(usage)  # None means the stream ended before the usage chunk
```

The SDK reads up to `[DONE]` for you. The [OpenAI Node SDK guide](https://broadrouter.com/guides/openai-node-sdk/) shows the same loop in JavaScript.

## When the stream ends early

If the connection closes before `[DONE]`, the answer is incomplete. BroadNet’s guidance:

-   record the `X-Request-ID`;
-   decide whether your application can safely make a **new** request. A retry is a new inference and is billed again on pay-as-you-go.

For agents on an [unlimited plan](https://broadrouter.com/guides/unlimited-llm-api-sessions/), retries cost nothing extra, but they still occupy the session while they run.

BroadNet’s docs say per-model release verification is still required before you rely on streaming in production, so test it on the model you ship with.

## Troubleshooting

-   **Everything prints at the end.** curl needs `-N`. Behind your own proxy, turn off response buffering for `text/event-stream`.
-   **`IndexError` or `undefined` on the last chunk.** The usage chunk has an empty `choices` list. Check `chunk.choices` before reading `[0]`.
-   **No usage at all.** You did not send `stream_options`, or the stream broke before the final chunk.
-   **Stream stops with `finish_reason` `length` and little text.** A reasoning model spent `max_tokens` on reasoning. See [why reasoning models return empty answers](https://broadrouter.com/guides/reasoning-tokens-empty-response/).
-   **429 while streaming in parallel.** Each open stream counts as a request in flight; a pay-as-you-go key allows 4. See [BroadNet error codes and retries](https://broadrouter.com/guides/errors-and-retries/).

## Terms in this guide

[Streaming](https://broadrouter.com/glossary/streaming/)

Streaming is a way of returning a language model response in small chunks while it is still being generated, instead of as one reply at the end. OpenAI-compatible APIs stream over server-sent events when the request sets stream to true.

[Pay-as-you-go](https://broadrouter.com/glossary/pay-as-you-go/)

Pay-as-you-go is a pricing model in which you pay only for what you use, metered per unit such as tokens, requests or audio minutes, with no subscription or minimum commitment. Most LLM APIs bill this way, from a prepaid balance or a periodic invoice.

[Server-sent events (SSE)](https://broadrouter.com/glossary/sse/)

Server-sent events (SSE) is a web standard for a server to push a stream of text events to a client over one long-lived HTTP response with the content type text/event-stream. Each event is one or more field lines, such as data:, ended by a blank line.

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[Cached input tokens](https://broadrouter.com/glossary/cached-input-tokens/)

Cached input tokens are prompt tokens that an LLM provider has recently processed and can reuse from a cache instead of computing again, usually a repeated prefix such as a system prompt or document. Providers bill them at a lower rate than new input tokens.

[Concurrency](https://broadrouter.com/glossary/concurrency/)

Concurrency is the number of requests a client has in flight at the same time. LLM APIs often cap it per key or plan, separately from requests per minute, because each open request holds GPU capacity until its response has finished.

## Questions

### How do I get token usage when streaming chat completions?

Send stream\_options with include\_usage set to true. One extra chunk arrives just before data: \[DONE\], with usage filled in and an empty choices array. If the stream breaks first, that chunk never arrives.

### Why does curl print the whole streamed answer at once?

curl buffers its output by default. Add the -N flag (no buffering) so each Server-Sent Event prints as it arrives.

### Is a retried stream billed twice?

On pay-as-you-go, yes. A retry is a new inference request. On an unlimited DeepSeek V4.1 Flash plan there is no token meter, but the retry still occupies the session while it runs.

### Does streaming cost more than a normal request?

No. Streaming changes how the answer is delivered, not how it is billed. On pay-as-you-go you pay the same per-token input and output prices as for a request without streaming.

## Related guides

-   [Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/)
-   [BroadNet Inference error codes and when to retry](https://broadrouter.com/guides/errors-and-retries/)
-   [Why a reasoning model returns an empty answer](https://broadrouter.com/guides/reasoning-tokens-empty-response/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=guides-streaming-chat-completions&utm_content=page-cta) ·

**Sources**

-   [BroadNet Inference API reference, streaming](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=guides-streaming-chat-completions&utm_content=sources#requests)
-   [WHATWG HTML Standard, server-sent events (checked 2026-11-14)](https://html.spec.whatwg.org/multipage/server-sent-events.html)
-   [OpenAI Python SDK, stream\_options include\_usage docstring (checked 2026-11-14)](https://github.com/openai/openai-python/blob/main/src/openai/types/chat/chat_completion_stream_options_param.py)
