Every time a model writes a token, it looks back at every token before it. The KV cache is what makes that look-back cheap.
How it works
Attention compares the current token’s query with a key and a value for each earlier token. Without a cache, the model would recompute every earlier key and value on each step. Hugging Face’s caching guide sums up the trade: with a cache, “attention cost per step is linear with sequence length”, while memory “grows linearly”.
So inference has two phases:
- Prefill. The model reads the whole prompt and fills the cache. This is most of the time to first token.
- Decode. Each new token adds one key and one value per layer, then reads the whole cache.
Cache size per token is roughly 2 × layers × KV heads × head size × bytes per number. Model designers attack every term. DeepSeek’s card for DeepSeek V4.1 Flash reports a global KV cache of 890 bytes per token, using FP4 storage. At the full 1,048,576-token context that is 890 × 1,048,576 = 933,232,640 bytes, about 890 MiB for one request.
Why it matters
- Concurrency and price. The PagedAttention paper says serving many requests at once is limited because KV cache memory “is huge and grows and shrinks dynamically”. vLLM pages that memory like an operating system and reports 2 to 4 times higher throughput at the same latency.
- Long context is not free. A provider has to hold the cache for your whole prompt while it generates. That is one reason long-context requests are slower to start.
- Prompt caching builds on it. Keeping a prefix’s KV state after a request ends is what a provider bills as cached input tokens. Stable prefixes hit that cache more often; see prompt structure for caching.
- Quantization applies here too. Storing keys and values in fewer bits shrinks the cache, at some risk to quality.
On BroadNet Inference
BroadNet does not publish how it manages KV cache memory. What its API reference does document is the visible effect: usage.prompt_tokens_details.cached_tokens reports prompt tokens read from cache, billed at the cached input price, for example $0.0055 per 1M tokens on DeepSeek V4.1 Flash against $0.175 for new input. That model and GLM 5.3 accept up to 1,048,576 tokens per request.
Common mistakes
- Treating context length as only a price question. A larger cache also means a slower prefill.
- Reordering a system prompt on each turn. The cached prefix no longer matches, so the provider recomputes it.