BroadRouter
Get started

KV cache

Updated

A KV cache is the memory where a transformer language model stores the attention keys and values computed for every token it has already processed, so each new token reuses them instead of recomputing the whole sequence. It grows with context length and often limits how many requests a GPU can serve.

Also called: key-value cache, past_key_values, attention cache, KV memory

Every time a model writes a token, it looks back at every token before it. The KV cache is what makes that look-back cheap.

How it works

Attention compares the current token’s query with a key and a value for each earlier token. Without a cache, the model would recompute every earlier key and value on each step. Hugging Face’s caching guide sums up the trade: with a cache, “attention cost per step is linear with sequence length”, while memory “grows linearly”.

So inference has two phases:

  1. Prefill. The model reads the whole prompt and fills the cache. This is most of the time to first token.
  2. Decode. Each new token adds one key and one value per layer, then reads the whole cache.

Cache size per token is roughly 2 × layers × KV heads × head size × bytes per number. Model designers attack every term. DeepSeek’s card for DeepSeek V4.1 Flash reports a global KV cache of 890 bytes per token, using FP4 storage. At the full 1,048,576-token context that is 890 × 1,048,576 = 933,232,640 bytes, about 890 MiB for one request.

Why it matters

  • Concurrency and price. The PagedAttention paper says serving many requests at once is limited because KV cache memory “is huge and grows and shrinks dynamically”. vLLM pages that memory like an operating system and reports 2 to 4 times higher throughput at the same latency.
  • Long context is not free. A provider has to hold the cache for your whole prompt while it generates. That is one reason long-context requests are slower to start.
  • Prompt caching builds on it. Keeping a prefix’s KV state after a request ends is what a provider bills as cached input tokens. Stable prefixes hit that cache more often; see prompt structure for caching.
  • Quantization applies here too. Storing keys and values in fewer bits shrinks the cache, at some risk to quality.

On BroadNet Inference

BroadNet does not publish how it manages KV cache memory. What its API reference does document is the visible effect: usage.prompt_tokens_details.cached_tokens reports prompt tokens read from cache, billed at the cached input price, for example $0.0055 per 1M tokens on DeepSeek V4.1 Flash against $0.175 for new input. That model and GLM 5.3 accept up to 1,048,576 tokens per request.

Common mistakes

  • Treating context length as only a price question. A larger cache also means a slower prefill.
  • Reordering a system prompt on each turn. The cached prefix no longer matches, so the provider recomputes it.

Questions

Is the KV cache the same as prompt caching?

Not quite. The KV cache lives inside one request while the model generates. Prompt caching keeps that state after the request ends so a later request with the same prefix can reuse it, and providers bill those tokens as cached input.

Why does a long prompt use so much GPU memory?

Every token in the context adds its keys and values to the cache, in every layer. A million-token prompt keeps a million tokens of attention state in memory until the reply finishes.

How big is a KV cache?

Roughly 2 × layers × KV heads × head size × bytes per number, for every token in the context. DeepSeek's card reports 890 bytes per token for DeepSeek V4.1 Flash, about 890 MiB at its full 1,048,576-token context.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity