Apple's recent research in artificial intelligence has offered new insights into the concept of "distillation," a technique gaining traction in the AI field. This approach involves transferring knowledge from larger, complex "teacher" models to smaller, simpler "student" models. The core advantage of this method lies in leveraging the outputs of powerful models to enhance the intelligence of smaller models more efficiently.
The study highlights that multiple rounds of distillation are beneficial, and the performance of the teacher model is more crucial than its size. A significant performance gap between the teacher and student models can hinder the distillation process, emphasizing the need for an appropriately matched teacher to facilitate learning.
Apple's research, conducted in collaboration with Oxford University, introduces a distillation scaling law that predicts the performance of distilled models based on computational budget distribution. Key findings include the relationship between data volume and training method, where supervised learning outperforms distillation given sufficient computational resources or tokens. However, when token budgets are limited, distillation proves advantageous.
Moreover, when training multiple student models from an existing teacher model, distillation is often more cost-effective. The performance level of the teacher model, measured by cross-entropy loss, is more important than its size. The study suggests selecting a teacher model slightly larger than the student model for optimal results.
Apple's research defines how student model performance depends on the teacher's cross-entropy loss, dataset size, and model parameters, offering a theoretical foundation for deciding when to distill a model or apply supervised fine-tuning.
