BroadRouter
Get started

Speculative decoding

Updated

Speculative decoding is an inference technique in which a small, fast draft model proposes several next tokens and the large target model verifies them in a single forward pass, keeping the accepted ones. It speeds up generation without changing the distribution of the output.

Also called: speculative sampling, assisted generation, assisted decoding, draft model decoding

A large model writes one token per forward pass. Speculative decoding lets it confirm several tokens per pass instead.

How it works

Leviathan et al. start from a plain fact: “decoding K tokens takes K serial runs of the model.” Their fix borrows speculative execution from CPUs:

  1. A draft model, much smaller and faster, guesses the next few tokens.
  2. The target model scores all of those guesses in one forward pass. Scoring a short run of tokens costs about the same as producing one, as Chen et al. observe.
  3. A modified rejection-sampling step keeps the longest run of guesses the target model agrees with, then samples one token of its own.
  4. Repeat from the new position.

When the draft is right often, each expensive pass yields several tokens. When it is wrong, you lose little: the target model still produces at least one token. Hugging Face’s guide adds that the method works best when the draft model is “significantly smaller” and shares the same tokenizer.

Variants drop the separate draft model. The same guide describes multi-token prediction (MTP) layers, where extra heads in the main checkpoint draft the candidates. DeepSeek’s model card for DeepSeek V4.1 Flash lists its own scheme, “DSpark speculative decoding”, as part of the architecture.

Why it matters

For an API caller, speculative decoding shows up as higher per-request tokens per second, which cuts total latency on long answers. It does not shorten time to first token, which is dominated by reading the prompt.

The gain is uneven. Predictable text such as code edits, JSON and boilerplate is easy to draft. Creative, high-temperature text is harder, so fewer guesses survive. Under heavy load the extra verification work competes with other requests, so a provider may tune it for per-user speed or for total throughput.

It is also a server-side choice. You cannot turn it on through a standard chat completions request.

On BroadNet Inference

BroadNet’s API reference does not mention speculative decoding. Its model catalogue lists the production Gemma 4 26B A4B build, offered on the API as BroadNet Gemma 4 VL 26B, as running with MTP (k=2), which the catalogue describes as a form of speculative decoding. BroadNet’s model page puts that build at about 136 tokens per second on a single stream. The catalogue also reports that EAGLE3 cut throughput by 69% at 32 concurrent requests, which is why BroadNet kept MTP drafters, and BroadNet has published two studies of Uno, a self-drafting adapter: Uno in vLLM and Uno on Gemma 4 26B. For its flat-price plans, which run DeepSeek V4.1 Flash, BroadNet’s plans page gives a typical speed of about 100 tokens per second. Measure your own output speed with streaming; the LLM latency guide shows how.

Questions

Does speculative decoding change the answer?

Not in distribution. Both original papers use an acceptance rule that keeps the target model's output distribution, so quality matches the large model on its own. Floating-point details can still differ slightly between runs.

How much faster is speculative decoding?

The original papers report 2 to 3 times faster decoding on T5-XXL and 2 to 2.5 times on a 70B Chinchilla model. Real gains depend on how often the draft model guesses right.

Can I turn on speculative decoding through an API?

Not in a standard chat completions request. It is a serving choice made by the provider. You see it only as faster output, so measure tokens per second on your own prompts.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity