---
title: "What is streaming in an LLM API? Server-sent events | BroadRouter"
description: "Streaming returns an LLM reply in chunks while it is generated. How server-sent events and [DONE] work, how to get usage, and what to do when a stream breaks."
url: https://broadrouter.com/glossary/streaming/
lang: en
updated: 2026-11-14
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Streaming

Updated 2026-11-14

Streaming is a way of returning a language model response in small chunks while it is still being generated, instead of as one reply at the end. OpenAI-compatible APIs stream over server-sent events when the request sets stream to true.

Also called: streaming responses, stream=true, token streaming

Without streaming, the client waits until the whole answer is finished. With it, text appears as the model writes it.

## How it works

Streaming uses [server-sent events (SSE)](https://broadrouter.com/glossary/sse/). The server answers with content type `text/event-stream` and keeps the connection open. MDN describes the format: lines of `field: value`, with messages “separated by a pair of newline characters.”

In the Chat Completions format, each event carries a chunk with a small `delta`:

```text
data: {"choices":[{"index":0,"delta":{"content":"Hel"}}]}

data: {"choices":[{"index":0,"delta":{"content":"lo"}}]}

data: {"choices":[],"usage":{"prompt_tokens":12,"completion_tokens":2}}

data: [DONE]
```

The client joins the deltas. The literal `data: [DONE]` line marks the end. The usage chunk appears only if the request asked for it with `stream_options: {"include_usage": true}`.

## Why it matters

-   **Perceived speed.** Users see the first words after the [time to first token](https://broadrouter.com/glossary/time-to-first-token/), not after the whole reply. After that, text flows at the model’s [tokens per second](https://broadrouter.com/glossary/tokens-per-second/).
-   **Long outputs.** Streaming keeps the connection active, which helps with proxies that close idle requests.
-   **Early action.** Agents and UIs can start parsing or rendering before the end.

There are costs. OpenAI notes that streaming makes moderation harder, because “partial completions may be more difficult to evaluate.” A stream can also break halfway, and a streamed reply holds a [concurrency](https://broadrouter.com/glossary/concurrency/) slot until it ends.

## On BroadNet Inference

BroadNet uses the standard chat-completions streaming shape on `https://inference.broadnet.ai/v1/chat/completions`, so the [Python](https://broadrouter.com/guides/openai-python-sdk/) and [Node](https://broadrouter.com/guides/openai-node-sdk/) SDKs parse it for you. With curl, add `-N` to turn off buffering:

```bash
curl -N https://inference.broadnet.ai/v1/chat/completions \
  -H "Authorization: Bearer $BROADNET_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"broadnet/gemma-4-26b-a4b","stream":true,"stream_options":{"include_usage":true},"messages":[{"role":"user","content":"Count to three."}]}'
```

Reasoning models can report `reasoning_tokens` and `cached_tokens` in the final usage chunk. If the connection closes before `[DONE]`, treat the answer as incomplete, record the `X-Request-ID` header, and decide whether a new request is safe. BroadNet’s docs say to verify streaming on the model you ship with before relying on it in production. Details are in the [streaming guide](https://broadrouter.com/guides/streaming-chat-completions/) and [errors and retries](https://broadrouter.com/guides/errors-and-retries/).

## Common mistakes

-   Treating a closed connection as a finished answer. Only `[DONE]` means complete.
-   Putting a buffering proxy in front of the client. The reply then arrives all at once.

## Questions

### Does streaming make the model faster?

No. Generation speed is the same. Streaming shows the first words sooner, so the reply feels faster, and lets code act on partial output.

### How do I get token usage in a stream?

In the Chat Completions format, send stream\_options with include\_usage set to true. The last chunk then carries usage and an empty choices list.

### Is a retried stream billed twice?

On a per-token key, usually yes. A retry is a new inference, billed again. On a flat plan with no token meter, it only costs time.

## See also

-   [Tokens per second](https://broadrouter.com/glossary/tokens-per-second/) (term)
-   [Concurrency](https://broadrouter.com/glossary/concurrency/) (term)
-   [OpenAI-compatible API](https://broadrouter.com/glossary/openai-compatible-api/) (term)
-   [Input and output tokens](https://broadrouter.com/glossary/input-and-output-tokens/) (term)
-   [Server-sent events (SSE)](https://broadrouter.com/glossary/sse/) (term)
-   [Time to first token (TTFT)](https://broadrouter.com/glossary/time-to-first-token/) (term)
-   [Stream chat completions and handle dropped connections](https://broadrouter.com/guides/streaming-chat-completions/) (guide)
-   [Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/) (guide)
-   [Use the OpenAI Node.js SDK with BroadNet Inference](https://broadrouter.com/guides/openai-node-sdk/) (guide)
-   [BroadNet Inference error codes and when to retry](https://broadrouter.com/guides/errors-and-retries/) (guide)
-   [Switching to an OpenAI-compatible API: a migration checklist](https://broadrouter.com/blog/openai-compatible-api-migration-checklist/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-streaming&utm_content=page-cta) ·

**Sources**

-   [MDN, using server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events)
-   [OpenAI, streaming API responses](https://developers.openai.com/api/docs/guides/streaming-responses)
-   [BroadNet Inference API reference, streaming](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-streaming&utm_content=sources#requests)
