BroadRouter
Get started

Distillation (knowledge distillation)

Updated

Distillation, or knowledge distillation, is a training method in which a smaller student model learns to reproduce the outputs of a larger teacher model, such as its probability distributions or generated answers, so the student keeps much of the teacher's quality at a lower cost to run.

Also called: knowledge distillation, model distillation, distilled model, teacher-student training

Big models are expensive to serve. Distillation moves what a big model knows into a small one.

How it works

Hinton, Vinyals and Dean described the modern form in 2015. The teacher’s output probabilities become “soft targets” for the student. A plain label says only “this is a 7”. A teacher’s probabilities also say it looks a little like a 1 and not at all like a 4, and that extra signal helps the student learn.

To expose it, they raise the softmax temperature. The paper notes that a higher T “produces a softer probability distribution over classes”. Their MNIST results:

Model Test errors
Large teacher net 67
Small net, trained on labels only 146
Same small net, distilled at T = 20 74

With LLMs, distillation often uses generated text instead of probabilities. The teacher writes answers, and the student is fine-tuned on them. The DeepSeek-R1 paper reports that the reasoning patterns of large models can guide smaller ones. Its model card lists six distilled models from 1.5B to 70B parameters, built on Qwen2.5 and Llama bases and fine-tuned on reasoning data generated by DeepSeek-R1.

Why it matters

  • Cost and speed. A smaller model needs less GPU memory and generates faster, which shows up in per-token prices and latency.
  • Narrower skill. A student is strong where it was trained and weaker elsewhere. Test it on your own tasks rather than trusting a teacher’s benchmark.
  • Stacking. Distillation pairs with quantization and LoRA in self-hosted setups.
  • Licensing. The teacher’s licence or API terms decide whether you may train on its outputs.

On BroadNet Inference

BroadNet’s API reference does not say whether any of its models were distilled, so do not assume it. Price differences between models do show why small models matter: BroadNet Gemma 4 VL 26B lists $0.22 per 1M output tokens against $3.46 for GLM 5.3. Whether the smaller model is good enough is a question for your own eval. The post on the best open model for coding agents compares the options by task.

Questions

What is the difference between distillation and quantization?

Distillation trains a new, smaller model. Quantization keeps the same model and stores its weights at lower precision. Both make inference cheaper, and they can be combined.

Is a distilled model as good as the teacher?

Usually not on every task, but it can come close on the tasks it was distilled for. In the MNIST test by Hinton, Vinyals and Dean, the distilled small net made 74 errors against the teacher's 67.

Is distilling another provider's model allowed?

Read the teacher model's licence or API terms before you train on its outputs. They differ on this point, and some restrict it.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity