What is Model Distillation? 

‍

Model distillation is a machine learning technique with which a smaller, simpler model (called the “student” model) is trained to replicate the behavior of a larger, more complex model (the “teacher” model). 

The hypothesis is that the student model will learn to mimic the predictions of the teacher model based on a target dataset. This can be achieved by measuring the difference between the responses of the student and the teacher model respectively. As this difference is minimized over training, the student model will become better at making the same predictions as the teacher.

Arcee AI's mission is to enable users and businesses to train cost-effective, secure, performant, and domain-specific Small Language Models (SLMs). To support this mission, we’ve launched DistillKit, an open-source research initiative focused on model distillation. Dive into our DistillKit announcement and explore the full technical paper to learn more.

‍

‍

‍