A large model writes one token per forward pass. Speculative decoding lets it confirm several tokens per pass instead.
How it works
Leviathan et al. start from a plain fact: “decoding K tokens takes K serial runs of the model.” Their fix borrows speculative execution from CPUs:
- A draft model, much smaller and faster, guesses the next few tokens.
- The target model scores all of those guesses in one forward pass. Scoring a short run of tokens costs about the same as producing one, as Chen et al. observe.
- A modified rejection-sampling step keeps the longest run of guesses the target model agrees with, then samples one token of its own.
- Repeat from the new position.
When the draft is right often, each expensive pass yields several tokens. When it is wrong, you lose little: the target model still produces at least one token. Hugging Face’s guide adds that the method works best when the draft model is “significantly smaller” and shares the same tokenizer.
Variants drop the separate draft model. The same guide describes multi-token prediction (MTP) layers, where extra heads in the main checkpoint draft the candidates. DeepSeek’s model card for DeepSeek V4.1 Flash lists its own scheme, “DSpark speculative decoding”, as part of the architecture.
Why it matters
For an API caller, speculative decoding shows up as higher per-request tokens per second, which cuts total latency on long answers. It does not shorten time to first token, which is dominated by reading the prompt.
The gain is uneven. Predictable text such as code edits, JSON and boilerplate is easy to draft. Creative, high-temperature text is harder, so fewer guesses survive. Under heavy load the extra verification work competes with other requests, so a provider may tune it for per-user speed or for total throughput.
It is also a server-side choice. You cannot turn it on through a standard chat completions request.
On BroadNet Inference
BroadNet’s API reference does not mention speculative decoding. Its model catalogue lists the production Gemma 4 26B A4B build, offered on the API as BroadNet Gemma 4 VL 26B, as running with MTP (k=2), which the catalogue describes as a form of speculative decoding. BroadNet’s model page puts that build at about 136 tokens per second on a single stream. The catalogue also reports that EAGLE3 cut throughput by 69% at 32 concurrent requests, which is why BroadNet kept MTP drafters, and BroadNet has published two studies of Uno, a self-drafting adapter: Uno in vLLM and Uno on Gemma 4 26B. For its flat-price plans, which run DeepSeek V4.1 Flash, BroadNet’s plans page gives a typical speed of about 100 tokens per second. Measure your own output speed with streaming; the LLM latency guide shows how.