Get the 2026 ML Training Cookbook | 52 recipes — GRPO, Flow Matching, World Models, and everything in between Download Now →
Knowledge Distillation
A model compression technique where a smaller student model learns to mimic the behavior of a larger, more capable teacher model.
Knowledge Distillation is a model compression technique in which a smaller, more efficient 'student' model is trained to mimic the behavior of a larger, more capable 'teacher' model. The teacher's knowledge — encoded in its output probability distributions (soft labels) rather than just its final predictions (hard labels) — contains rich information about the relationships between classes. By learning from these soft labels, the student can achieve performance approaching the teacher despite having far fewer parameters.
The standard distillation process involves applying a temperature parameter to the teacher's softmax function, which softens the output distribution and reveals more of the 'dark knowledge' — the relative probabilities the teacher assigns to incorrect classes. The student is trained to match these softened distributions using a distillation loss (typically KL divergence), combined with a standard task loss on hard labels. The temperature controls how much emphasis is placed on the teacher's soft labels versus the ground-truth labels.
Knowledge distillation is widely used to deploy large models in resource-constrained environments. A massive teacher model that requires expensive GPU inference can be distilled into a compact student that runs on edge devices or serves high-throughput inference at low latency. Variants include on-policy distillation (where the student generates its own inputs), feature-based distillation (matching intermediate representations), and multi-teacher distillation.
- ■Student model learns from teacher's soft labels (softened probability distributions)
- ■Temperature parameter controls the softness of the teacher's output distribution
- ■Trained with a combination of distillation loss (KL divergence) and task loss
- ■Enables deployment of large model capabilities on resource-constrained hardware
- ■Variants include on-policy, feature-based, and multi-teacher distillation
Distilling a 70B-parameter teacher model into a 7B-parameter student that retains 95% of the teacher's accuracy while running 10× faster at inference.