BroadRouter

Uno in vLLM: what BroadNet's serving study measured

By the BroadNet Inference team · 4 min read · Published

Uno in vLLM is a September 2026 preprint by BroadNet (DOI 10.5281/zenodo.22652610). It adds Uno, an adapter that lets a model draft its own tokens, to vLLM. On an H100 with Qwen3-8B, BroadNet measured 2.58 times plain-vLLM throughput at one stream, 2.20 times at eight and 1.74 times at 32, within 3% of the reference implementation.

Uno in vLLM is a preprint from BroadNet’s research arm, published on Zenodo on 8 September 2026 under DOI 10.5281/zenodo.22652610. It reports a native vLLM implementation of Uno, a speculative-decoding method developed by IFM, and measures how much faster it serves than plain vLLM. Its Zenodo record lists Mohammad Al Zubaidi as the author.

This page summarises what the paper says, how it measured it, and what it does and does not tell you. The numbers are BroadNet’s; we have not reproduced them.

What problem does Uno address?

A language model normally generates one token per forward pass. Speculative decoding speeds this up by guessing several tokens ahead and checking them in one pass of the full model. Most methods use a separate, smaller draft model. Uno instead attaches a trained adapter to the model itself, so the model drafts its own tokens and then verifies them with its original weights. See speculative decoding for the general idea.

The paper’s question is practical: does that speed-up survive inside a production serving engine, with batching, caching and many requests at once?

What did BroadNet build?

BroadNet’s blog post is clear about credit: “Uno, developed by IFM and collaborators” is the method, and BroadNet’s contribution is “the native vLLM integration, the engineering that makes its execution paths work together, and the serving evaluation”. The abstract lists the execution pieces: conditional adapter routing, complete draft key/value state, asynchronous scheduling, CUDA graph replay, overlapping adapter computation and seed-row reuse.

The code is public in BroadNet’s vllm-uno repository, and BroadNet has proposed it to vLLM upstream as pull request 55947.

How was it measured?

The main test is a controlled one:

  • Hardware and model: one NVIDIA H100, Qwen3-8B in bfloat16, with the published Qwen3-8B Uno adapter.
  • Workload: eight math prompts, cycled to fill larger batches, each generating up to 512 tokens, with prefix caching off.
  • Baseline: ordinary autoregressive vLLM (“plain vLLM”) on the same setup.
  • Concurrency: 1, 8 and 32 simultaneous streams.

A second, separate comparison turns prefix caching on and uses BroadNet’s chosen serving settings, to compare against the SGLang reference implementation.

What did the study find?

Measurement Result reported by BroadNet
Throughput vs plain vLLM, 1 stream 2.58x
Throughput vs plain vLLM, 8 streams 2.20x
Throughput vs plain vLLM, 32 streams 1.74x
Gap to the SGLang reference implementation within 3% at 1, 8 and 32 streams
Production settings, 32 streams 7,657 tokens/s, against 7,505 for SGLang (2.0% ahead)
Warm first-token latency, within-vLLM ablation 25% lower with two API processes instead of one

The headline for capacity planning is the last row of the first block: 74% more output per second at 32 concurrent streams, from the same GPU. That is what BroadNet’s blog calls “the central payoff”, because a speed-up that disappears under load does little for a shared API.

A later release (v0.2.0, on vLLM’s Model Runner V2) is described on BroadNet’s blog with 2.1 to 2.4 times single-stream output on Blackwell, 1.7 to 1.9 times on Ampere and 2.6 times on Hopper. Those are different hardware and versions from the paper’s main table, which is why the figures differ.

What are the limits?

  • One model, one workload. The controlled result is Qwen3-8B on math prompts. Code, chat or long documents can accept fewer drafted tokens.
  • Gains shrink with load. The trend from 2.58x to 1.74x is the expected shape: when the GPU is busy, spare compute for drafting runs out.
  • A preprint. The paper is deposited on Zenodo, not peer-reviewed. Its companion archive holds the evidence inventory and figure data, which is what lets others check it.
  • Self-reported. BroadNet ran both arms. The 3% agreement with the reference implementation is the paper’s own check on that.

Why does it matter if you just call an API?

Throughput per GPU sets what a provider can charge and how fast it answers under load. BroadNet’s catalogue reports that it measures drafting methods at real concurrency before choosing one, and that it ruled out EAGLE3 after it cost 69% of throughput at 32 concurrent requests. The follow-up study on Gemma 4 26B applies the same method to the mixture-of-experts model BroadNet sells as Gemma 4 VL 26B. For how this fits BroadNet’s wider serving choices, see BroadNet’s GPU hardware and serving approach.

If you run your own vLLM servers, the practical lesson is to benchmark any drafting method at the concurrency you actually run, not at one stream.

Questions

What is Uno?

Uno is a speculative-decoding method developed by IFM and collaborators. A small trained adapter lets the model propose several tokens at once, and the model's own weights then verify them. There is no separate draft model.

Is the speed-up lossless?

BroadNet says so for its release: whenever plain vLLM reproduces its own output, Uno matches its greedy tokens, and its sampled-distribution tests pass. Speculative decoding verifies every drafted token with the full model, so output quality should not change; check this on your own prompts.

Does Uno help at high concurrency?

Less than at low concurrency. In the paper's controlled test the gain fell from 2.58x at one stream to 1.74x at 32. BroadNet's catalogue also reports that another method, EAGLE3, cost 69% of throughput at 32 concurrent requests, so drafting methods need testing at your real load.

Can I run it myself?

Yes. BroadNet publishes the implementation in its vllm-uno repository, with a container image, pinned versions and a validation record for each release, and has proposed it to vLLM as pull request 55947.

Does BroadNet Inference use Uno for its API models?

BroadNet does not say so. Its catalogue lists multi-token-prediction drafters (k=2) in production for Gemma 4 26B. Uno is a research result BroadNet publishes; ask BroadNet if you need it on a private endpoint.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity