---
title: "What are prefill and decode? The two phases of LLM inference"
description: "Prefill reads your whole prompt at once; decode then writes the answer one token at a time. How the two phases differ and why they set TTFT and speed."
url: https://broadrouter.com/glossary/prefill-and-decode/
lang: en
updated: 2026-11-16
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Prefill and decode

Updated 2026-11-16

Prefill and decode are the two phases of LLM inference. In prefill the model processes the whole prompt in parallel and fills its KV cache; in decode it generates the reply one token at a time, reusing that cache. Prefill mostly sets time to first token; decode sets output speed.

Also called: prefill, prefill phase, decode phase, prefill stage, prompt processing

A chat request does two different jobs: read the prompt, then write the answer. Inference engineers call them prefill and decode, and they behave very differently on a GPU.

## How it works

1.  **Prefill.** The model reads every prompt token in one parallel pass and stores each token’s attention keys and values in the [KV cache](https://broadrouter.com/glossary/kv-cache/). Nothing is returned yet.
2.  **Decode.** The model generates one token, appends its keys and values to the cache, and repeats until it stops or hits `max_tokens`.

The Sarathi-Serve paper puts the contrast plainly: “Prefill iterations have high latency but saturate GPU compute due to parallel processing of the input prompt”, while “decode iterations have low latency but also low compute utilization.”

The two phases map onto the two latency numbers users feel. The DistServe paper names them: “time to first token (TTFT) for the prefill phase and time per output token (TPOT) of each request for the decoding phase.”

A worked example: a request with a 20,000-token prompt and a 300-token answer does all 20,000 tokens of prefill before the first streamed token appears, then 300 sequential decode steps. Cutting the prompt in half shortens the wait for the first token; it does not change how fast the 300 tokens arrive.

## Why it matters

-   **[Time to first token](https://broadrouter.com/glossary/time-to-first-token/) tracks prompt size.** Long system prompts, documents and tool lists all add prefill work. Cached prefixes skip some of it, which is why [cached input tokens](https://broadrouter.com/glossary/cached-input-tokens/) can make long prompts start faster.
-   **[Tokens per second](https://broadrouter.com/glossary/tokens-per-second/) is a decode number.** Output speed depends on how quickly each decode step runs, which [speculative decoding](https://broadrouter.com/glossary/speculative-decoding/) tries to raise.
-   **The phases interfere.** When a server runs both on the same GPU, a big prefill can pause other users’ decodes. DistServe reports “strong prefill-decoding interferences” and moves the phases to different GPUs. Sarathi-Serve instead “splits a prefill request into near equal sized chunks” and schedules them “without pausing ongoing decodes”.
-   **Engines expose the trade-off.** vLLM’s tuning guide says chunked prefill lets it batch prefill chunks with decode requests. A smaller `max_num_batched_tokens` improves inter-token latency, while a larger one favours TTFT and [throughput](https://broadrouter.com/glossary/throughput/).

The [latency, TTFT and throughput guide](https://broadrouter.com/blog/llm-latency-ttft-throughput/) shows how to measure both phases from the client side.

## On BroadNet Inference

BroadNet does not publish how its servers schedule prefill and decode. What you can see is the effect: stream a response as in the [streaming guide](https://broadrouter.com/guides/streaming-chat-completions/), log the time to the first content chunk (mostly prefill plus queueing) and the rate of chunks after it (decode). On [GLM 5.3](https://broadrouter.com/models/glm-5.3/), [DeepSeek V4.1 Flash](https://broadrouter.com/models/deepseek-v4.1-flash/) and [Qwen3.8 27B](https://broadrouter.com/models/qwen3.8-27b/), reasoning tokens are generated in decode too, so they delay the visible answer and count toward `max_tokens`.

## Common mistakes

-   Blaming a slow first token on the model’s output speed. Check prompt length first.
-   Benchmarking with a tiny prompt. Real prompts with long context have a much longer prefill.

## Questions

### Why does a long prompt make the first token slow?

Every prompt token has to go through prefill before the model can write anything. A longer prompt means more prefill work, so time to first token grows with prompt length unless part of the prompt is served from cache.

### Does a longer answer slow down the first token?

No. The first token waits on prefill of the prompt. A longer answer only adds more decode steps after the first token has arrived.

### What is chunked prefill?

A scheduling method that splits a long prompt into pieces and mixes them with ongoing decode steps, so one large prompt does not stall other users. vLLM enables it by default where it can.

## See also

-   [KV cache](https://broadrouter.com/glossary/kv-cache/) (term)
-   [Time to first token (TTFT)](https://broadrouter.com/glossary/time-to-first-token/) (term)
-   [Tokens per second](https://broadrouter.com/glossary/tokens-per-second/) (term)
-   [Throughput](https://broadrouter.com/glossary/throughput/) (term)
-   [Latency](https://broadrouter.com/glossary/latency/) (term)
-   [vLLM](https://broadrouter.com/glossary/vllm/) (term)
-   [Stream chat completions and handle dropped connections](https://broadrouter.com/guides/streaming-chat-completions/) (guide)
-   [LLM latency: TTFT vs tokens per second vs end-to-end time](https://broadrouter.com/blog/llm-latency-ttft-throughput/) (article)
-   [Self-hosting an LLM vs an API: the real cost and break-even](https://broadrouter.com/blog/self-hosting-llm-vs-api-cost/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-prefill-and-decode&utm_content=page-cta) ·

**Sources**

-   [Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving, arXiv 2401.09670 (checked 2026-11-15)](https://arxiv.org/abs/2401.09670)
-   [Agrawal et al., Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve, arXiv 2403.02310 (checked 2026-11-15)](https://arxiv.org/abs/2403.02310)
-   [vLLM docs, optimization and tuning: chunked prefill (checked 2026-11-15)](https://docs.vllm.ai/en/latest/configuration/optimization/)
