Big models are expensive to serve. Distillation moves what a big model knows into a small one.
How it works
Hinton, Vinyals and Dean described the modern form in 2015. The teacher’s output probabilities become “soft targets” for the student. A plain label says only “this is a 7”. A teacher’s probabilities also say it looks a little like a 1 and not at all like a 4, and that extra signal helps the student learn.
To expose it, they raise the softmax temperature. The paper notes that a higher T “produces a softer probability distribution over classes”. Their MNIST results:
| Model | Test errors |
|---|---|
| Large teacher net | 67 |
| Small net, trained on labels only | 146 |
| Same small net, distilled at T = 20 | 74 |
With LLMs, distillation often uses generated text instead of probabilities. The teacher writes answers, and the student is fine-tuned on them. The DeepSeek-R1 paper reports that the reasoning patterns of large models can guide smaller ones. Its model card lists six distilled models from 1.5B to 70B parameters, built on Qwen2.5 and Llama bases and fine-tuned on reasoning data generated by DeepSeek-R1.
Why it matters
- Cost and speed. A smaller model needs less GPU memory and generates faster, which shows up in per-token prices and latency.
- Narrower skill. A student is strong where it was trained and weaker elsewhere. Test it on your own tasks rather than trusting a teacher’s benchmark.
- Stacking. Distillation pairs with quantization and LoRA in self-hosted setups.
- Licensing. The teacher’s licence or API terms decide whether you may train on its outputs.
On BroadNet Inference
BroadNet’s API reference does not say whether any of its models were distilled, so do not assume it. Price differences between models do show why small models matter: BroadNet Gemma 4 VL 26B lists $0.22 per 1M output tokens against $3.46 for GLM 5.3. Whether the smaller model is good enough is a question for your own eval. The post on the best open model for coding agents compares the options by task.