AI Definitions: Knowledge Distillation
/Knowledge distillation (KD) - A machine learning technique that transfers the learnings of a large pretrained model to a smaller model. While the “student model” will mimic the predictions of the big one, it is more agile and efficient, able to make better real-time decisions than a large model. The smaller model can also more easily include in its structure “explainability” (reasoning behind the decisions). KD is often used in deep learning and particularly for deep neural networks.