---
title: "What is a KV cache? How LLMs reuse attention work | BroadRouter"
description: "A KV cache stores attention keys and values for tokens an LLM has already read, so each new token is cheap to compute. How it works and why it limits long context."
url: https://broadrouter.com/glossary/kv-cache/
lang: en
updated: 2026-11-14
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# KV cache

Updated 2026-11-14

A KV cache is the memory where a transformer language model stores the attention keys and values computed for every token it has already processed, so each new token reuses them instead of recomputing the whole sequence. It grows with context length and often limits how many requests a GPU can serve.

Also called: key-value cache, past\_key\_values, attention cache, KV memory

Every time a model writes a token, it looks back at every token before it. The KV cache is what makes that look-back cheap.

## How it works

Attention compares the current token’s query with a key and a value for each earlier token. Without a cache, the model would recompute every earlier key and value on each step. Hugging Face’s caching guide sums up the trade: with a cache, “attention cost per step is linear with sequence length”, while memory “grows linearly”.

So inference has two phases:

1.  **Prefill.** The model reads the whole prompt and fills the cache. This is most of the [time to first token](https://broadrouter.com/glossary/time-to-first-token/).
2.  **Decode.** Each new token adds one key and one value per layer, then reads the whole cache.

Cache size per token is roughly `2 × layers × KV heads × head size × bytes per number`. Model designers attack every term. DeepSeek’s card for DeepSeek V4.1 Flash reports a global KV cache of 890 bytes per token, using FP4 storage. At the full 1,048,576-token context that is 890 × 1,048,576 = 933,232,640 bytes, about 890 MiB for one request.

## Why it matters

-   **Concurrency and price.** The PagedAttention paper says serving many requests at once is limited because KV cache memory “is huge and grows and shrinks dynamically”. vLLM pages that memory like an operating system and reports 2 to 4 times higher [throughput](https://broadrouter.com/glossary/throughput/) at the same latency.
-   **Long context is not free.** A provider has to hold the cache for your whole prompt while it generates. That is one reason long-context requests are slower to start.
-   **Prompt caching builds on it.** Keeping a prefix’s KV state after a request ends is what a provider bills as [cached input tokens](https://broadrouter.com/glossary/cached-input-tokens/). Stable prefixes hit that cache more often; see [prompt structure for caching](https://broadrouter.com/blog/prompt-structure-for-caching/).
-   **[Quantization](https://broadrouter.com/glossary/quantization/) applies here too.** Storing keys and values in fewer bits shrinks the cache, at some risk to quality.

## On BroadNet Inference

BroadNet does not publish how it manages KV cache memory. What its API reference does document is the visible effect: `usage.prompt_tokens_details.cached_tokens` reports prompt tokens read from cache, billed at the cached input price, for example $0.0055 per 1M tokens on [DeepSeek V4.1 Flash](https://broadrouter.com/models/deepseek-v4.1-flash/) against $0.175 for new input. That model and GLM 5.3 accept up to 1,048,576 tokens per request.

## Common mistakes

-   Treating context length as only a price question. A larger cache also means a slower prefill.
-   Reordering a system prompt on each turn. The cached prefix no longer matches, so the provider recomputes it.

## Questions

### Is the KV cache the same as prompt caching?

Not quite. The KV cache lives inside one request while the model generates. Prompt caching keeps that state after the request ends so a later request with the same prefix can reuse it, and providers bill those tokens as cached input.

### Why does a long prompt use so much GPU memory?

Every token in the context adds its keys and values to the cache, in every layer. A million-token prompt keeps a million tokens of attention state in memory until the reply finishes.

### How big is a KV cache?

Roughly 2 × layers × KV heads × head size × bytes per number, for every token in the context. DeepSeek's card reports 890 bytes per token for DeepSeek V4.1 Flash, about 890 MiB at its full 1,048,576-token context.

## See also

-   [Cached input tokens](https://broadrouter.com/glossary/cached-input-tokens/) (term)
-   [Context window](https://broadrouter.com/glossary/context-window/) (term)
-   [Time to first token (TTFT)](https://broadrouter.com/glossary/time-to-first-token/) (term)
-   [Throughput](https://broadrouter.com/glossary/throughput/) (term)
-   [Quantization](https://broadrouter.com/glossary/quantization/) (term)
-   [Speculative decoding](https://broadrouter.com/glossary/speculative-decoding/) (term)
-   [Prefill and decode](https://broadrouter.com/glossary/prefill-and-decode/) (term)
-   [vLLM](https://broadrouter.com/glossary/vllm/) (term)
-   [Stream chat completions and handle dropped connections](https://broadrouter.com/guides/streaming-chat-completions/) (guide)
-   [When a 1M-token context window is worth it (and when not)](https://broadrouter.com/blog/million-token-context-window-guide/) (article)
-   [How to structure prompts so the prefix caches](https://broadrouter.com/blog/prompt-structure-for-caching/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-kv-cache&utm_content=page-cta) ·

**Sources**

-   [Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (vLLM), arXiv 2309.06180](https://arxiv.org/abs/2309.06180)
-   [Hugging Face Transformers, caching explained](https://huggingface.co/docs/transformers/en/cache_explanation)
-   [DeepSeek, DeepSeek-V4.1-Flash model card (KV cache bytes per token)](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash)
